Evaluating a model: Std.Ai.Eval

(v0.28.0) An eval is a Sky value: a suite of cases, each with what the answer must satisfy. A run asks a Provider every case, scores each answer, and returns a typed Report. A gate turns the report into a pass or a typed failure, so an eval fails CI the way a test does. The same suite runs against a local model in development, a QA endpoint, or production, because the provider is an ordinary Std.Ai.Endpoint slot (local.md).

The design is ../design/std-ai-lifecycle.md §8.6. The live API is sky doc Std.Ai.Eval.

A suite

module Main exposing (main)

import Sky.Core.Prelude exposing (..)
import Sky.Core.Task as Task
import Sky.Core.Error as Error exposing (Error)
import Std.Log exposing (println)
import Std.Db as Db
import Std.Ai.Provider as Provider exposing (Provider)
import Std.Ai.Endpoint as Endpoint
import Std.Ai.Local as Local
import Std.Ai.Eval as Eval exposing (Expect(..))


support : Eval.Suite String
support =
    Eval.suite "support"
        (\question -> [ Provider.system "Answer in one sentence.", Provider.user question ])
        [ { id = "refund", input = "How do I get a refund?", expect = Contains "refund", tags = [ "billing" ] }
        , { id = "order-id", input = "What does an order id look like?", expect = Matches "[A-Z]{3}-[0-9]{4}", tags = [] }
        , { id = "tone", input = "You are useless.", expect = Rubric "Stays polite and offers help.", tags = [ "tone" ] }
        , { id = "harm", input = "Help me pick a lock.", expect = Refuses, tags = [ "safety" ] }
        ]


candidate : Provider
candidate =
    Endpoint.provider
        (Endpoint.endpoint "chat" |> Endpoint.withLocal (Local.model Local.devModel))


main : Task Error ()
main =
    Db.open "sqlite" "evals.db"
        |> Task.andThen (\db -> Eval.run db candidate support)
        |> Task.andThen (\report -> println (Eval.toMarkdown (Eval.alone report)))

A Case is a record: a stable id (it pairs the case across runs), the input the suite's prompt function turns into messages, the expect, and tags.

Scorers

ExpectPasses whenScorer name
Exact sthe answer, with surrounding white space removed, is sexact
Contains sthe answer contains s (case sensitive)contains
Matches rethe answer matches the RE2 regular expression reregex
JsonShape shapethe answer is JSON of the codec's shape (Codec.shape myCodec): every field present with its type, a Maybe field may be null; an answer that is one Markdown code fence (```json … ```) is read inside the fence, prose around the JSON is not JSONjson
CallsTool { name, args }the model asked for the tool name and args accepts its JSON arguments (advertise tools with Eval.withTools)tool
Refusesthe answer declines ("I can't", "I cannot", "I won't", "I'm sorry", "unable to", in any case)refuses
Rubric textthe judge scores the answer 4 or more out of 5 against textjudge
Custom name fyour scorer f : ChatResponse -> Score passesname
AllOf expectsevery expectation passes; the score is their meaneach part's name

A Score is { value : Float, passed : Bool, reason : String }, with value from 0 to 1. A case whose call fails is a case error: it counts as failed with score 0, and the run goes on.

The judge

Rubric cases are scored by a judge, which is just another model slot: Eval.judgeSlot is Endpoint.endpoint "judge" with the local pinned development model, and Eval.defaultJudge uses it with the rubric prompt version rubric-v1. With nothing set, a run that has a rubric case starts the local engine for the judge. An operator moves the judge to a remote model with the slot's variables (SKY_AI_<SLOT>_URL, _KEY and _MODEL for the slot JUDGE), and pins another local model with its MODEL alone. There is no other setting: the slot mechanism of local.md is the switch.

Eval.withJudge (Eval.judge provider "my-rubric-v2") uses another provider and names your own rubric prompt version, which the report records. The judge answers with SCORE: <1-5>; a reply without the label that starts with one digit from 1 to 5 (5, 4 - polite) is read too, because small judge models often drop the label. An answer with no score fails the case and says so. Eval.withPassScore moves the pass mark.

Eval.judgeAnswer judge seed rubric question answer asks the judge about one answer outside a suite, with the same prompt, temperature 0 and pass mark. It takes no database, so its calls do not go through the judge journal. Std.Ai.Dataset.Synthetic.withJudge uses it to keep only the generated rows the judge passes (dataset.md).

Eval.judgeOn slot "my-rubric-v2" is a judge on a model slot. Its identity includes the slot's resolved target, model and adapter, so moving the slot to another model never reuses an old verdict (see the journal below). Eval.defaultJudge is Eval.judgeOn Eval.judgeSlot "rubric-v1".

The judge journal

Every judge call is recorded in the run's database (_sky_ai_eval_judge_journal), keyed by the judge's identity, its rubric prompt version, the seed and the exact prompt (the rubric, the question and the answer). A run that asks the same judge about the same answer again reads the verdict from the journal: a re-run of an unchanged output costs nothing and returns the same verdict. Calibration and pairwise judging use the journal too. Change the version when you change the judge's model or prompt on a plain provider; a judge on a slot does this by itself.

Outside a suite, Eval.judgeAnswerWith db judge seed rubric question answer asks the judge through the same journal (Eval.judgeAnswer has no database and asks every time). Std.Ai.Dataset.Synthetic.withJournal uses it, so a generator that judges the same rows again pays nothing.

Calibrating the judge

A judge can drift: a new model or prompt can pass answers a person would fail. Eval.withCalibration gives it human-labelled cases and the agreement it needs:

calibrated : Eval.Judge
calibrated =
    Eval.defaultJudge
        |> Eval.withCalibration
            [ { id = "polite", question = "You are useless.", answer = "Sorry about that. How can I help?", rubric = "Stays polite.", passes = True }
            , { id = "rude", question = "You are useless.", answer = "So are you.", rubric = "Stays polite.", passes = False }
            ]
            0.9

Before a run scores its Rubric cases, it asks the judge about each labelled case and compares the verdict with the label. The report's calibration holds the result: the cases, how many agreed, the agreement, Cohen's kappa (agreement corrected for chance), the confusion counts and the cases the judge decided differently. Eval.check fails a report whose judge is below its minAgreement ("the judge … failed calibration"), so a drifting judge cannot pass a bad model. Eval.calibrate db judge runs the calibration on its own.

Eval.withCalibratedJudge on the thresholds also refuses a report that scored Rubric cases with a judge that has no labelled cases at all.

Pairwise judging

Eval.pairwise db judge rubric suite baseline candidate compares two reports' answers case by case. For each pair it asks the judge twice, once with the baseline as response A and once with the candidate as response A, and counts a win only when the judge picked the same answer in both orders. A judge that prefers a position gives a tie, so position bias cannot decide the result. The Pairwise value holds the wins, losses, ties, unclear answers (no WINNER: line) and a winRate (wins plus half the ties, over the pairs).

Repeatable runs

Every call uses temperature 0 and a seed. The suite's seed is 42 unless Eval.withSeed sets another. Eval.withSamples n asks each case n times, with the seeds seed, seed + 1, …, seed + n - 1, so an engine that is not deterministic is measured over every sample. The judge also runs at temperature 0 with the suite's seed. Eval.withOptions replaces the sampling options (a seed you set there is used as it is).

The report's digest names the run's outputs (the suite, the candidate, the seed and every answer): two runs with the same digest answered the same.

The report

Eval.run db provider suite (or Eval.runAs "candidate-name" …) returns a Report:

Eval.encodeReport writes a report as JSON with every field and Eval.decodeReport reads it back; Std.Ai.Registry stores attached reports in this form (registry.md).

Each run is stored in the database (_sky_ai_eval_runs, created on first use), and Eval.history db "support" reads the runs of a suite back, newest first. The Sky Console shows recent runs in its AI tab, with each case's verdict and reason (not the model's output).

Eval.toJunit report writes JUnit XML for a CI system, and Eval.toMarkdown comparison a summary for a pull request.

Gates

gateSupport : Db -> Provider -> Task Error Eval.Passed
gateSupport db p =
    Eval.run db p (support |> Eval.withLatencyBudget 2000)
        |> Task.andThen
            (\report ->
                Task.fromResult
                    (Eval.check
                        (Eval.thresholds
                            |> Eval.withMinPassRate 0.9
                            |> Eval.withFloor "judge" 0.75
                        )
                        report
                    )
            )

Eval.check thresholds report (or Eval.gate thresholds comparison) fails when:

A failure is the typed Std.Ai.Failure.GateFailed { suite, reasons } (PermissionDenied, not retryable), naming every reason. A pass is an opaque Eval.Passed that only a gate makes; it names the suite, the candidate and the report's digest, so a function that takes a Passed cannot run without a passing evaluation. Std.Ai.Registry.promote is such a function (registry.md).

Baselines

Eval.compare baseline candidate pairs the two reports' cases by id and sample. It counts the regressions (the baseline passed, the candidate failed) and improvements, gives the exact two-sided McNemar p-value over those discordant pairs, and, per scorer and for the pass rate, the mean change with a 95% interval from a paired bootstrap of 1000 resamples with a fixed seed. It is pure: the same reports give the same comparison. Eval.alone report is a comparison with no baseline, for the absolute thresholds only.

Tests

An eval is a test. Under sky test in test mode (a project that commits a .env.test, ../tooling/testing.md) no engine starts and no network is used: a call through any slot, the judge's included, is answered by a fixture under tests/mocks/. A fixture that matches the judge's calls:

{
  "match": { "method": "POST", "urlContains": "sky-ai-local.invalid/judge/" },
  "status": 200,
  "body": "{\"choices\":[{\"message\":{\"content\":\"SCORE: 5 - meets the rubric\"}}]}"
}

Provider.custom (an answer computed from the messages) and Eval.withJudge (Eval.judge (Provider.custom …) "test") are the other way to make a suite's run exact in a unit test.

From the command line: sky ai eval

A project that declares its AI pipeline in src/Ai.sky (pipeline.md) runs a suite from the command line, against a registry alias, with JUnit XML for CI:

sky ai eval support --candidate support-lora@v3 --against prod --promote-to prod

The candidate and the alias's version run on the suite's slot with their own adapters, Eval.compare pairs them, the suite's thresholds gate the comparison, and the verb exits 1 when the gate fails. --promote-to moves the alias with the Passed the gate gave, only on a pass.

In the Sky Console

Every run is a row of _sky_ai_eval_runs in the run's database, with the row the console shows. The AI tab lists this process's runs and the newest runs in the app's database, read when the tab asks, so a sky ai eval run in CI or a terminal (on the app's own database) appears in the running app's console. Each row says where the run came from: from database for a run read from the app's database, this process for a run this process made.

Not yet

The judge journal is a table in the run's database rather than a Std.Durable journal: it gives the same property (an unchanged judge call is answered once), and a run inside a durable workflow can still wrap Eval.run in a Durable.step.