Std.Ai.Eval
Std.Ai.Eval — evaluate a model with Sky values: suites of cases, scorers,
Types
type alias Case input = { id : String , input : input , expect : Expect , tags : List String }
type Expect = Exact String | Contains String | Matches String | JsonShape Codec.Shape | CallsTool { name : String, args : String -> Bool } | Refuses | Rubric String | Custom String (ChatResponse -> Score) | AllOf (List Expect)
type alias Score = { value : Float , passed : Bool , reason : String }
type Suite input
type Judge
type alias Labelled = { id : String , question : String , answer : String , rubric : String , passes : Bool }
type alias CaseResult = { id : String , sample : Int , tags : List String , passed : Bool , score : Float , scores : List ( String, Float ) , reason : String , output : String , latencyMs : Int , cost : Maybe Money , error : Maybe String }
type alias ScorerScore = { scorer : String , mean : Float , count : Int }
type alias Report = { suite : String , candidate : String , seed : Int , samples : Int , judge : Maybe String , startedAt : Int , durationMs : Int , cases : List CaseResult , passed : Int , failed : Int , errors : Int , passRate : Float , scorers : List ScorerScore , p95LatencyMs : Int , meanCost : Maybe Money , latencyBudgetMs : Maybe Int , costBudget : Maybe Money , calibration : Maybe Calibration , digest : String , traceRun : String }
type alias Calibration = { judge : String , cases : Int , agreed : Int , agreement : Float , kappa : Float , truePass : Int , trueFail : Int , falsePass : Int , falseFail : Int , minAgreement : Float , calibrated : Bool , disagreements : List Disagreement }
type alias Disagreement = { id : String , label : Bool , verdict : Bool , reason : String }
type alias RunSummary = { digest : String , suite : String , candidate : String , passed : Int , failed : Int , errors : Int , passRate : Float , p95LatencyMs : Int , createdAt : Int }
type alias JudgeCtx = { judge : Judge , db : Db , identity : String , trace : Maybe TraceTo }
type alias TraceTo = { runId : String , judgeBook : Maybe PriceBook }
type alias CaseInfo = { id : String , tags : List String , expect : Expect }
type alias Output = { content : String , toolCalls : List ToolCall , usage : Provider.Usage }
type alias Called = { result : Result Error Output , record : Task Error () }
type alias PairResult = { id : String , sample : Int , winner : String , first : String , second : String }
type alias Pairwise = { suite : String , baseline : String , candidate : String , judge : String , pairs : Int , candidateWins : Int , baselineWins : Int , ties : Int , unclear : Int , winRate : Float , cases : List PairResult }
type alias ScorerDelta = { scorer : String , baseline : Float , candidate : Float , delta : Float , ciLow : Float , ciHigh : Float , pairs : Int }
type alias Comparison = { candidate : Report , baseline : Maybe Report , deltas : List ScorerDelta , regressions : Int , improvements : Int , mcnemarP : Float }
type Thresholds
type Passed
Values
alone : Report -> Comparison
A candidate on its own: the absolute thresholds apply, the regression
calibrate : Db -> Judge -> Task Error Calibration
Calibrate a judge on its labelled cases (`withCalibration`): ask the
check : Thresholds -> Report -> Result Error Passed
`gate` for a report on its own.
compare : Report -> Report -> Comparison
Compare a baseline (first) and a candidate (second), pairing cases by
decodeReport : String -> Result Error Report
A report from the JSON `encodeReport` wrote. A report written before a
defaultJudge : Judge
The default judge: `judgeSlot` with the rubric prompt `rubric-v1`.
encodeReport : Report -> String
A report as JSON text: every field, so `decodeReport` reads the same
gate : Thresholds -> Comparison -> Result Error Passed
Apply the thresholds. It fails when the pass rate or a scorer's mean is
history : Db -> String -> Task Error (List RunSummary)
The runs of a suite in the database, newest first.
judge : Provider -> String -> Judge
A judge on a provider, with the version name of your rubric prompt. The
judgeAnswer : Judge -> Int -> String -> String -> String -> Task Error Score
Ask the judge to score one answer against a rubric, outside a suite: the
judgeAnswerWith : Db -> Judge -> Int -> String -> String -> String -> Task Error Score
`judgeAnswer` through the judge journal in a database: an unchanged call
judgeOn : Endpoint -> String -> Judge
A judge on a model slot. Its identity in the judge journal is the
judgeSlot : Endpoint
The judge's model slot, `"judge"`: the local pinned development model by
pairwise : Db -> Judge -> String -> Suite input -> Report -> Report -> Task Error Pairwise
Compare a baseline report (first) and a candidate report (second) with
passedCandidate : Passed -> String
The candidate the passing report named (`Registry.label v` for a
passedDigest : Passed -> String
The digest of the report that passed.
passedSuite : Passed -> String
The suite the passing report ran.
prompts : Suite input -> List { messages : List Message, options : Provider.Options }
Every request a run of the suite sends, in order: each case's messages
run : Db -> Provider -> Suite input -> Task Error Report
Run a suite against a provider, record it in the database and the
runAs : String -> Db -> Provider -> Suite input -> Task Error Report
`run`, naming the candidate (a model, an adapter, a registry version
setup : Db -> Task Error ()
Create or update the eval tables (idempotent): the run history and the
suite : String -> (input -> List Message) -> List (Case input) -> Suite input
A suite: its name, how an input becomes messages, and its cases. One
suiteName : Suite input -> String
The suite's name: the `suite` field of its reports.
thresholds : Thresholds
No pass-rate floor, no scorer floors, and an allowed regression of 0.05
toJunit : Report -> String
JUnit XML for a CI system: one test case per case and sample, a failure
toMarkdown : Comparison -> String
A Markdown summary of a comparison, for a pull request or a CI log.
withCalibratedJudge : Thresholds -> Thresholds
Require a calibrated judge: a report that scored `Rubric` cases with a
withCalibration : List Labelled -> Float -> Judge -> Judge
Calibrate the judge on human-labelled cases. Before a run scores any
withConcurrency : Int -> Suite input -> Suite input
How many calls run at once (at least one).
withCostBudget : Money -> Suite input -> Suite input
The mean cost per case. A report over it fails the gate. Needs
withFloor : String -> Float -> Thresholds -> Thresholds
The lowest mean of one scorer (`"exact"`, `"judge"`, a custom name, or
withJudge : Judge -> Suite input -> Suite input
The judge for `Rubric` cases. Without it, `defaultJudge`.
withJudgePriceBook : PriceBook -> Suite input -> Suite input
Price the judge's calls in `Std.Ai.Trace`. Without it the judge's rows
withLatencyBudget : Int -> Suite input -> Suite input
The p95 latency budget in milliseconds. A report over it fails the gate.
withMaxRegression : Float -> Thresholds -> Thresholds
The largest allowed drop of a scorer against the baseline, measured at
withMinPassRate : Float -> Thresholds -> Thresholds
The lowest pass rate, from 0 to 1.
withOptions : Provider.Options -> Suite input -> Suite input
The sampling options. The suite's seed fills `seed` when it is `Nothing`.
withPassScore : Int -> Judge -> Judge
The lowest score from 1 to 5 that passes (default 4).
withPriceBook : PriceBook -> Suite input -> Suite input
Price each call, for the report's cost figures and `withCostBudget`.
withSamples : Int -> Suite input -> Suite input
Ask each case `n` times (at least once), with the seeds `seed` to
withSeed : Int -> Suite input -> Suite input
The base seed (default 42).
withTools : List ToolSpec -> Suite input -> Suite input
Advertise tools, for `CallsTool` cases. The calls then go through