The model registry: Std.Ai.Registry

(v0.28.0) The registry holds the versions of your fine-tuned adapters, what each was trained from, the eval reports that gate it, and the aliases (dev, qa, prod) that serving follows. Promotion needs a passing evaluation, and the type system enforces it: Registry.promote takes an Eval.Passed, which only an eval gate can make (eval.md).

The design is ../design/std-ai-lifecycle.md §8.7. The live API is sky doc Std.Ai.Registry and sky doc Std.Ai.Store. examples/71-ai-registry runs the whole lifecycle with no model and no network. examples/73-ai-train trains an adapter with Std.Ai.Train, registers it and serves it (train.md).

Two halves: the records and the artefacts

Only adapters are registered: LoRA and QLoRA, as GGUF (what llama.cpp loads) or safetensors (what training and vLLM use). register refuses a file that is not one (a full model, a pickle checkpoint), and each version is pinned to the digest of the base model it was trained on.

Register, evaluate, promote

module Main exposing (main)

import Sky.Core.Prelude exposing (..)
import Sky.Core.Task as Task
import Sky.Core.Error as Error exposing (Error)
import Sky.Core.Crypto as Crypto
import Std.Log exposing (println)
import Std.Db as Db exposing (Db)
import Std.Ai.Provider as Provider exposing (Provider)
import Std.Ai.Local as Local
import Std.Ai.Eval as Eval exposing (Expect(..))
import Std.Ai.Store as Store
import Std.Ai.Registry as Registry exposing (Format(..), Method(..))


support : Eval.Suite String
support =
    Eval.suite "support"
        (\q -> [ Provider.user q ])
        [ { id = "refund", input = "How do I get a refund?", expect = Contains "refund", tags = [] } ]


-- The candidate: the base model with the new adapter (a slot that serves it).
candidate : Provider
candidate =
    Provider.custom (\_ -> Task.succeed { content = "Ask for a refund.", usage = { promptTokens = 1, completionTokens = 1, totalTokens = 2 } })


release : Db -> Registry.Registry -> Task Error Registry.AliasEvent
release db reg =
    Registry.register reg "support-lora"
        { path = "out/adapter.gguf", format = Gguf, method = Lora }
        { base = Local.devModel
        , dataset = "support@3"
        , datasetDigest = Crypto.sha256 "the frozen dataset's bytes"
        , metadata = [ ( "seed", "7" ), ( "backend", "mlx" ) ]
        }
        |> Task.andThen
            (\v ->
                Eval.runAs (Registry.label v) db candidate support
                    |> Task.andThen (\report -> Registry.attachReport reg v report |> Task.map (\_ -> report))
                    |> Task.andThen (\report -> Task.fromResult (Eval.check (Eval.thresholds |> Eval.withMinPassRate 1.0) report))
                    |> Task.andThen (\passed -> Registry.promote reg "prod" v passed)
            )


main : Task Error ()
main =
    Db.connect ()
        |> Task.andThen (\db -> Registry.openConfigured db "file://.sky/ai/registry" |> Task.andThen (release db))
        |> Task.andThen (\ev -> println ("prod now v" ++ String.fromInt ev.to))

Registry.promoteChecked reg "prod" v thresholds does the evaluation part from the evidence already attached: it re-applies the gate to the version's attached reports, newest first (Registry.evidence). This is what sky ai promote runs.

From a training run

A Std.Ai.Train job with Train.withRegistry reg registers its adapter when the run finishes, as the workflow's last journalled step (train.md):

supportJob : Registry.Registry -> Dataset.Version Dataset.Chat -> Train.Job
supportJob reg version =
    Train.lora Train.devBase version
        |> Train.withName "support"
        |> Train.withQuantisedBase
        |> Train.withGgufExport
        |> Train.withRegistry reg

The adapter's registered field is the label (support@v1). A resumed run registers the same bytes, which return the same version. Train.registryAdapter and Train.lineage build the same values for your own Registry.register call. Promotion stays a separate step: promote still needs an Eval.Passed for the trained version.

Aliases: promote, roll back, force

Every alias move is a compare-and-swap with a logged event, in one transaction:

An alias is lower case ([a-z][a-z0-9_-]*) and may not look like a version number (v3), so support-lora@v3 and support-lora@prod are never ambiguous.

Reading

FunctionGives
resolve reg "support-lora@prod" / "support-lora@v3"the version
version reg name n, versions reg nameone version, every version (newest first)
aliases reg nameevery alias with its version and generation
history reg name aliasthe alias's moves, newest first
reports reg v, evalHistory reg namethe reports of a version, of every version
lineage reg refthe version, the aliases on it now, its reports and the moves to or from it
trainedOn reg datasetevery version trained on a dataset, named by its digest or as <name>@<n> (the ones to retrain after an erasure)
datasetVersion reg "support@3", datasetVersions reg namea registered dataset version, every version of a dataset
fetchDataset reg da verified local path of a dataset version's JSONL
fetch reg va verified local path of the adapter

versionValue, lineageValue, eventValue and reportLinkValue give the same records as JSON.

Dataset versions

registerDataset reg manifest dataPath registers a frozen dataset version (Std.Ai.Dataset.freeze, dataset.md): the data file goes into the store under its digest (it must match the manifest's, ChecksumMismatch otherwise), and the version is a row of _sky_ai_registry_datasets named <name>@<n> with its manifest. Registering the same digest again returns the version already there. A training lineage records datasetLabel d and d.digest, which is what trainedOn and sky ai lineage --dataset search.

Serving follows the aliases

A model slot serves a registry adapter when it has a resolver and its adapter names a registry reference:

chatSlot : Endpoint
chatSlot =
    Endpoint.endpoint "chat"
        |> Endpoint.withLocal (Local.model Local.devModel)
        |> Endpoint.fromRegistry (Registry.resolver registry)
        |> Endpoint.withAdapter "registry:support-lora@prod"

where registry : Task Error Registry.Registry opens the registry (the slot opens it on its first call, so the slot stays a pure top-level value). The operator can set the adapter instead, with the slot's SKY_AI_<SLOT>_ADAPTER (for this slot, CHAT) set to registry:support-lora@qa.

A slot name names one slot in a process: give two registries' adapters two slot names.

Companions

A version is one file (its digest is the version's identity), and it can carry companion files, each stored by digest and recorded in the version's metadata under its role:

Registry.registerWith reg "support" adapter lineage
    [ { role = "peft", path = "run/adapter/adapter_model.safetensors" }
    , { role = "peft.config", path = "run/adapter/adapter_config.json" }
    ]

Std.Ai.Train registers a trained adapter this way: the GGUF file is the version (what llama.cpp loads) and the PEFT weights and adapter_config.json are its companions (what vLLM loads). A safetensors version can carry its GGUF export as gguf. Registry.companion "peft" v reads a digest back.

Pointer objects: what a serving tier reads

The records live in the app's database, and a GPU tier must not need that database. So the registry also publishes, to its store:

ObjectHoldsWritten
models/<name>/versions/<n>.jsonthe version's manifest: served name, digests, format, the base's id and digest, the companions' digestsonce, by register
aliases/<name>/<alias>.json{ number, generation, digest, servedName }by every alias move, after the database commits

An alias pointer is replaced only by a newer generation, through the store's conditional write (GCS's object generation, S3's ETag, a local file's digest), so a delayed writer can never move it back. Registry.readPointer, Registry.readManifest and Registry.storeResolver (a slot resolver that reads only these objects) use them; Std.Ai.Serve reads them to follow the aliases. A publish that the store refuses fails the call, saying that the database change stands and naming the repair:

sky ai registry publish support      # write every manifest and pointer of a model again

The artefact store

Store.configured "file://.sky/ai/registry" takes the location from code and lets the operator override it with SKY_AI_REGISTRY (under the project's prefix). Registry.openConfigured db url is Store.configured then open.

LocationStoreCredentials
file://<dir>Store.local dirnone
gs://<bucket>/<prefix>Store.gcs bucket prefixthe service-account key file GOOGLE_APPLICATION_CREDENTIALS, else the metadata server of GCE, GKE or Cloud Run
s3://<bucket>/<prefix>?region=<r>Store.s3 region bucket prefixAWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN, else the instance role through IMDSv2

Credentials come from the platform, never from a Sky variable. An upload is a conditional create (GCS ifGenerationMatch=0, S3 If-None-Match: *): a blob is never overwritten, and an upload of a blob that is already there succeeds. A downloaded blob is kept in this machine's artefact cache (under the AI home) after its digest is checked. Store.withEndpoint points a store at an emulator or an S3-compatible server (path-style URLs), with Store.withToken or Store.withAwsKeys for its credentials. S3 requests are signed with AWS Signature Version 4, written in Sky (Store.signV4, tested against the AWS examples).

The CLI

The registry verbs open the registry on the app's database. In a project with a pipeline (src/Ai.sky) that is the database its code names (Pipeline.withDb); --db <dsn> wins over it, and without a pipeline the database is --db, else SKY_DB_PATH, else DATABASE_URL. The store is --registry <url> (which wins over the variable), else SKY_AI_REGISTRY, else the pipeline's withRegistry. Nothing is read from sky.toml. Each takes --format json.

sky ai promote support-lora 3 --to prod                 # on the attached evidence
sky ai promote support-lora 3 --to prod --min-pass-rate 0.95
sky ai promote support-lora 3 --to prod --expect 2      # only while prod is v2
sky ai promote support-lora 3 --to qa --force --reason "manual QA build"
sky ai rollback support-lora@prod
sky ai lineage support-lora@prod
sky ai lineage --dataset <sha256>                       # versions trained on a dataset
sky ai lineage --dataset support@3                      # the same, by registered version

sky ai promote runs promoteCompared: the attached evidence is compared against the version the alias serves now, so a regression against production is refused even when the candidate's own pass rate clears the floor, and a promotion that raced it fails with PromotionConflict instead of overwriting it. In a project with an AI pipeline (pipeline.md) each suite's report is gated with that suite's thresholds from the pipeline. sky ai eval … --candidate <model>@v<n> --against <alias> --promote-to <alias> evaluates, compares and promotes in one verb.

A refused promotion exits non-zero with the typed failure: NotPromotable (no attached report, a report for another version, a blocked version), GateFailed (the attached reports do not pass the thresholds, or regress against the alias's version), or PromotionConflict (--expect did not match, or the alias moved).

Failures

FailureWhenErrorKind
NotRegistered { ref }an unknown version or aliasNotFound
NotPromotable { ref, reason }the evidence does not name the version, is not attached, or the version is blocked; nothing to roll back toPermissionDenied
PromotionConflict { ref, expected, actual }the alias moved since the caller lookedConflict
AdapterMismatch { adapter, adapterBase, engineBase }a local engine runs another base model than the adapter was trained onInvalidInput
InvalidSettinga file that is not an adapter, an unpinned base, a bad name or aliasInvalidInput
ChecksumMismatcha stored or downloaded artefact does not hash to its nameConflict

The console

The Sky Console's AI tab shows each registry model this process read or changed: its aliases, and on a click its versions, its alias log and its eval history (../observability.md).