← all modules

Std.Ai.Pipeline

Std.Ai.Pipeline

Types
  type Pipeline
  type alias Context = { name : String , db : Db , registry : Task Error Registry.Registry , args : List String }
  type DatasetSpec
  type Target
  type SuiteSpec
  type alias EvalRequest = { candidate : Maybe String , against : Maybe String , minPassRate : Maybe Float , maxRegression : Maybe Float }
  type alias EvalOutcome = { suite : String , candidate : Eval.Report , baseline : Maybe Eval.Report , comparison : Eval.Comparison , verdict : Result Error Eval.Passed , version : Maybe Registry.Version , baselineVersion : Maybe Registry.Version }
  type alias Frozen = { manifest : Dataset.Manifest , path : String , registered : Maybe Registry.DatasetVersion }
  type alias Members = { name : String , datasets : List String , suites : List String , slots : List String , tasks : List String }
  type alias Run = { label : String , provider : Provider , version : Maybe Registry.Version }

Values
  commandLine : Pipeline -> Task Error ()
      The program behind the project-mode `sky ai` verbs: `main` of the
  commandLineWith : Pipeline -> List String -> Task Error ()
      `commandLine` with the arguments given (a test drives it this way).
  context : Pipeline -> List String -> Task Error Context
      Open the pipeline's database and make the context its members get.
  datasetJson : (Context -> Task Error (Dataset.Splits String)) -> DatasetSpec
      A dataset of JSON rows (as `Dataset.fromJsonLines` reads them).
  datasetOf : Codec row -> (Context -> Task Error (Dataset.Splits row)) -> DatasetSpec
      A dataset of typed rows: the task builds its split rows (from files,
  evalAgainst : Pipeline -> String -> String -> Task Error Eval.Passed
      Evaluate a suite against an alias's version (`"prod"`) and fail with
  evalRequest : EvalRequest
      The default request: the slot as configured, alone.
  evalSuite : Pipeline -> String -> Task Error Eval.Passed
      Evaluate a suite on its own and fail with `GateFailed` when the gate
  evaluate : Pipeline -> String -> EvalRequest -> Task Error EvalOutcome
      Evaluate a suite: the candidate, and with `against` the alias's version
  forModel : String -> SuiteSpec -> SuiteSpec
      The registry model the suite gates: `--against prod` compares with
  freezeDataset : Pipeline -> String -> Task Error Frozen
      Freeze a dataset under `.sky/ai/datasets/<name>` (or its `withFolder`)
  members : Pipeline -> Members
      The names of the pipeline's members.
  outcomeValue : EvalOutcome -> JsonE.Value
      An outcome as the one JSON object `sky ai eval --format json` prints.
  pipeline : String -> Pipeline
      A pipeline named `name`, on the app's own database (`Db.connect ()`) and
  promote : Pipeline -> String -> Int -> String -> Task Error Registry.AliasEvent
      Move an alias to a registry version on its attached evidence, compared
  runTask : Pipeline -> String -> List String -> Task Error ()
      Run a pipeline task with arguments.
  suiteOn : Endpoint -> Eval.Suite input -> SuiteSpec
      A suite that runs on a slot. A registry candidate or baseline runs on
  suiteWith : (String -> Provider) -> Eval.Suite input -> SuiteSpec
      A suite whose provider you give for each adapter: the function gets
  warmSlot : Pipeline -> String -> Task Error Endpoint.Description
      Start a pipeline slot's engine with its code defaults and return its
  withDataset : String -> DatasetSpec -> Pipeline -> Pipeline
      A dataset `sky ai data freeze <name>` freezes.
  withDb : Task Error Db -> Pipeline -> Pipeline
      The database the eval runs, the judge journal and the registry records
  withFolder : (Dataset.Folder -> Dataset.Folder) -> DatasetSpec -> DatasetSpec
      Change where the dataset is frozen and what its manifest records:
  withRegistered : DatasetSpec -> DatasetSpec
      Register each frozen version in the registry
  withRegistry : String -> Pipeline -> Pipeline
      The registry's artefact store in code (`file://`, `gs://`, `s3://`).
  withSlot : Endpoint -> Pipeline -> Pipeline
      A model slot the pipeline names: `sky ai start <slot>` warms it with its
  withSuite : String -> SuiteSpec -> Pipeline -> Pipeline
      An eval suite `sky ai eval <name>` runs.
  withTask : String -> (Context -> Task Error ()) -> Pipeline -> Pipeline
      A task `sky ai run <name> -- <args>` runs: importing an adapter,
  withThresholds : Eval.Thresholds -> SuiteSpec -> SuiteSpec
      The suite's gate (default `Eval.thresholds`). `--min-pass-rate` and