← all modules

Std.Ai.Eval

Std.Ai.Eval
  Std.Ai.Eval — evaluate a model with Sky values: suites of cases, scorers,

Types
  type alias Case input = { id : String , input : input , expect : Expect , tags : List String }
  type Expect = Exact String | Contains String | Matches String | JsonShape Codec.Shape | CallsTool { name : String, args : String -> Bool } | Refuses | Rubric String | Custom String (ChatResponse -> Score) | AllOf (List Expect)
  type alias Score = { value : Float , passed : Bool , reason : String }
  type Suite input
  type Judge
  type alias Labelled = { id : String , question : String , answer : String , rubric : String , passes : Bool }
  type alias CaseResult = { id : String , sample : Int , tags : List String , passed : Bool , score : Float , scores : List ( String, Float ) , reason : String , output : String , latencyMs : Int , cost : Maybe Money , error : Maybe String }
  type alias ScorerScore = { scorer : String , mean : Float , count : Int }
  type alias Report = { suite : String , candidate : String , seed : Int , samples : Int , judge : Maybe String , startedAt : Int , durationMs : Int , cases : List CaseResult , passed : Int , failed : Int , errors : Int , passRate : Float , scorers : List ScorerScore , p95LatencyMs : Int , meanCost : Maybe Money , latencyBudgetMs : Maybe Int , costBudget : Maybe Money , calibration : Maybe Calibration , digest : String , traceRun : String }
  type alias Calibration = { judge : String , cases : Int , agreed : Int , agreement : Float , kappa : Float , truePass : Int , trueFail : Int , falsePass : Int , falseFail : Int , minAgreement : Float , calibrated : Bool , disagreements : List Disagreement }
  type alias Disagreement = { id : String , label : Bool , verdict : Bool , reason : String }
  type alias RunSummary = { digest : String , suite : String , candidate : String , passed : Int , failed : Int , errors : Int , passRate : Float , p95LatencyMs : Int , createdAt : Int }
  type alias JudgeCtx = { judge : Judge , db : Db , identity : String , trace : Maybe TraceTo }
  type alias TraceTo = { runId : String , judgeBook : Maybe PriceBook }
  type alias CaseInfo = { id : String , tags : List String , expect : Expect }
  type alias Output = { content : String , toolCalls : List ToolCall , usage : Provider.Usage }
  type alias Called = { result : Result Error Output , record : Task Error () }
  type alias PairResult = { id : String , sample : Int , winner : String , first : String , second : String }
  type alias Pairwise = { suite : String , baseline : String , candidate : String , judge : String , pairs : Int , candidateWins : Int , baselineWins : Int , ties : Int , unclear : Int , winRate : Float , cases : List PairResult }
  type alias ScorerDelta = { scorer : String , baseline : Float , candidate : Float , delta : Float , ciLow : Float , ciHigh : Float , pairs : Int }
  type alias Comparison = { candidate : Report , baseline : Maybe Report , deltas : List ScorerDelta , regressions : Int , improvements : Int , mcnemarP : Float }
  type Thresholds
  type Passed

Values
  alone : Report -> Comparison
      A candidate on its own: the absolute thresholds apply, the regression
  calibrate : Db -> Judge -> Task Error Calibration
      Calibrate a judge on its labelled cases (`withCalibration`): ask the
  check : Thresholds -> Report -> Result Error Passed
      `gate` for a report on its own.
  compare : Report -> Report -> Comparison
      Compare a baseline (first) and a candidate (second), pairing cases by
  decodeReport : String -> Result Error Report
      A report from the JSON `encodeReport` wrote. A report written before a
  defaultJudge : Judge
      The default judge: `judgeSlot` with the rubric prompt `rubric-v1`.
  encodeReport : Report -> String
      A report as JSON text: every field, so `decodeReport` reads the same
  gate : Thresholds -> Comparison -> Result Error Passed
      Apply the thresholds. It fails when the pass rate or a scorer's mean is
  history : Db -> String -> Task Error (List RunSummary)
      The runs of a suite in the database, newest first.
  judge : Provider -> String -> Judge
      A judge on a provider, with the version name of your rubric prompt. The
  judgeAnswer : Judge -> Int -> String -> String -> String -> Task Error Score
      Ask the judge to score one answer against a rubric, outside a suite: the
  judgeAnswerWith : Db -> Judge -> Int -> String -> String -> String -> Task Error Score
      `judgeAnswer` through the judge journal in a database: an unchanged call
  judgeOn : Endpoint -> String -> Judge
      A judge on a model slot. Its identity in the judge journal is the
  judgeSlot : Endpoint
      The judge's model slot, `"judge"`: the local pinned development model by
  pairwise : Db -> Judge -> String -> Suite input -> Report -> Report -> Task Error Pairwise
      Compare a baseline report (first) and a candidate report (second) with
  passedCandidate : Passed -> String
      The candidate the passing report named (`Registry.label v` for a
  passedDigest : Passed -> String
      The digest of the report that passed.
  passedSuite : Passed -> String
      The suite the passing report ran.
  prompts : Suite input -> List { messages : List Message, options : Provider.Options }
      Every request a run of the suite sends, in order: each case's messages
  run : Db -> Provider -> Suite input -> Task Error Report
      Run a suite against a provider, record it in the database and the
  runAs : String -> Db -> Provider -> Suite input -> Task Error Report
      `run`, naming the candidate (a model, an adapter, a registry version
  setup : Db -> Task Error ()
      Create or update the eval tables (idempotent): the run history and the
  suite : String -> (input -> List Message) -> List (Case input) -> Suite input
      A suite: its name, how an input becomes messages, and its cases. One
  suiteName : Suite input -> String
      The suite's name: the `suite` field of its reports.
  thresholds : Thresholds
      No pass-rate floor, no scorer floors, and an allowed regression of 0.05
  toJunit : Report -> String
      JUnit XML for a CI system: one test case per case and sample, a failure
  toMarkdown : Comparison -> String
      A Markdown summary of a comparison, for a pull request or a CI log.
  withCalibratedJudge : Thresholds -> Thresholds
      Require a calibrated judge: a report that scored `Rubric` cases with a
  withCalibration : List Labelled -> Float -> Judge -> Judge
      Calibrate the judge on human-labelled cases. Before a run scores any
  withConcurrency : Int -> Suite input -> Suite input
      How many calls run at once (at least one).
  withCostBudget : Money -> Suite input -> Suite input
      The mean cost per case. A report over it fails the gate. Needs
  withFloor : String -> Float -> Thresholds -> Thresholds
      The lowest mean of one scorer (`"exact"`, `"judge"`, a custom name, or
  withJudge : Judge -> Suite input -> Suite input
      The judge for `Rubric` cases. Without it, `defaultJudge`.
  withJudgePriceBook : PriceBook -> Suite input -> Suite input
      Price the judge's calls in `Std.Ai.Trace`. Without it the judge's rows
  withLatencyBudget : Int -> Suite input -> Suite input
      The p95 latency budget in milliseconds. A report over it fails the gate.
  withMaxRegression : Float -> Thresholds -> Thresholds
      The largest allowed drop of a scorer against the baseline, measured at
  withMinPassRate : Float -> Thresholds -> Thresholds
      The lowest pass rate, from 0 to 1.
  withOptions : Provider.Options -> Suite input -> Suite input
      The sampling options. The suite's seed fills `seed` when it is `Nothing`.
  withPassScore : Int -> Judge -> Judge
      The lowest score from 1 to 5 that passes (default 4).
  withPriceBook : PriceBook -> Suite input -> Suite input
      Price each call, for the report's cost figures and `withCostBudget`.
  withSamples : Int -> Suite input -> Suite input
      Ask each case `n` times (at least once), with the seeds `seed` to
  withSeed : Int -> Suite input -> Suite input
      The base seed (default 42).
  withTools : List ToolSpec -> Suite input -> Suite input
      Advertise tools, for `CallsTool` cases. The calls then go through