Std.Ai.Pipeline
Types
type Pipeline
type alias Context = { name : String , db : Db , registry : Task Error Registry.Registry , args : List String }
type DatasetSpec
type Target
type SuiteSpec
type alias EvalRequest = { candidate : Maybe String , against : Maybe String , minPassRate : Maybe Float , maxRegression : Maybe Float }
type alias EvalOutcome = { suite : String , candidate : Eval.Report , baseline : Maybe Eval.Report , comparison : Eval.Comparison , verdict : Result Error Eval.Passed , version : Maybe Registry.Version , baselineVersion : Maybe Registry.Version }
type alias Frozen = { manifest : Dataset.Manifest , path : String , registered : Maybe Registry.DatasetVersion }
type alias Members = { name : String , datasets : List String , suites : List String , slots : List String , tasks : List String }
type alias Run = { label : String , provider : Provider , version : Maybe Registry.Version }
Values
commandLine : Pipeline -> Task Error ()
The program behind the project-mode `sky ai` verbs: `main` of the
commandLineWith : Pipeline -> List String -> Task Error ()
`commandLine` with the arguments given (a test drives it this way).
context : Pipeline -> List String -> Task Error Context
Open the pipeline's database and make the context its members get.
datasetJson : (Context -> Task Error (Dataset.Splits String)) -> DatasetSpec
A dataset of JSON rows (as `Dataset.fromJsonLines` reads them).
datasetOf : Codec row -> (Context -> Task Error (Dataset.Splits row)) -> DatasetSpec
A dataset of typed rows: the task builds its split rows (from files,
evalAgainst : Pipeline -> String -> String -> Task Error Eval.Passed
Evaluate a suite against an alias's version (`"prod"`) and fail with
evalRequest : EvalRequest
The default request: the slot as configured, alone.
evalSuite : Pipeline -> String -> Task Error Eval.Passed
Evaluate a suite on its own and fail with `GateFailed` when the gate
evaluate : Pipeline -> String -> EvalRequest -> Task Error EvalOutcome
Evaluate a suite: the candidate, and with `against` the alias's version
forModel : String -> SuiteSpec -> SuiteSpec
The registry model the suite gates: `--against prod` compares with
freezeDataset : Pipeline -> String -> Task Error Frozen
Freeze a dataset under `.sky/ai/datasets/<name>` (or its `withFolder`)
members : Pipeline -> Members
The names of the pipeline's members.
outcomeValue : EvalOutcome -> JsonE.Value
An outcome as the one JSON object `sky ai eval --format json` prints.
pipeline : String -> Pipeline
A pipeline named `name`, on the app's own database (`Db.connect ()`) and
promote : Pipeline -> String -> Int -> String -> Task Error Registry.AliasEvent
Move an alias to a registry version on its attached evidence, compared
runTask : Pipeline -> String -> List String -> Task Error ()
Run a pipeline task with arguments.
suiteOn : Endpoint -> Eval.Suite input -> SuiteSpec
A suite that runs on a slot. A registry candidate or baseline runs on
suiteWith : (String -> Provider) -> Eval.Suite input -> SuiteSpec
A suite whose provider you give for each adapter: the function gets
warmSlot : Pipeline -> String -> Task Error Endpoint.Description
Start a pipeline slot's engine with its code defaults and return its
withDataset : String -> DatasetSpec -> Pipeline -> Pipeline
A dataset `sky ai data freeze <name>` freezes.
withDb : Task Error Db -> Pipeline -> Pipeline
The database the eval runs, the judge journal and the registry records
withFolder : (Dataset.Folder -> Dataset.Folder) -> DatasetSpec -> DatasetSpec
Change where the dataset is frozen and what its manifest records:
withRegistered : DatasetSpec -> DatasetSpec
Register each frozen version in the registry
withRegistry : String -> Pipeline -> Pipeline
The registry's artefact store in code (`file://`, `gs://`, `s3://`).
withSlot : Endpoint -> Pipeline -> Pipeline
A model slot the pipeline names: `sky ai start <slot>` warms it with its
withSuite : String -> SuiteSpec -> Pipeline -> Pipeline
An eval suite `sky ai eval <name>` runs.
withTask : String -> (Context -> Task Error ()) -> Pipeline -> Pipeline
A task `sky ai run <name> -- <args>` runs: importing an adapter,
withThresholds : Eval.Thresholds -> SuiteSpec -> SuiteSpec
The suite's gate (default `Eval.thresholds`). `--min-pass-rate` and