Fine-tuning adapters: Std.Ai.Train

(v0.28.0) Std.Ai.Train fine-tunes a LoRA or QLoRA adapter from Sky code. A job trains a base model on a frozen dataset version (dataset.md). The run is a durable workflow: the trainer runs in a detached process, a trainer that dies is relaunched from its newest checkpoint, and the finished adapter is converted to GGUF and registered with its lineage (registry.md). The managed local engine then serves it (local.md).

The trainer needs no Python. Mlx trains with sky-trainer, Sky's native LoRA trainer (C on MLX through mlx-c), installed from a pinned, hash-checked bundle. Fake trains in the runtime itself. Only Unsloth (and Vertex, which runs the same driver in a container) still runs a Python driver, in a pinned interpreter with a hash-locked set of wheels that Sky installs under the AI home; you never install or see Python.

The design is ../design/std-ai-lifecycle.md §8.5 (the shipped notes in §14 P3 record where this release differs from it). The live API is sky doc Std.Ai.Train. examples/73-ai-train is a web app that trains, registers and serves an adapter, and its tests run the whole lifecycle offline.

Quick start

From the command line

sky ai provision --dev-model              # the engine and the Q4_0 dev model
sky ai provision --train mlx --base dev   # the native trainer and the 4-bit base weights
sky ai data freeze data/support.jsonl --name support
sky ai train --data .sky/ai/datasets/support --steps 60 --gguf

sky ai train trains the newest frozen version of support on the development base and prints the adapter, its losses and the GGUF file. A Ctrl-C stops the watcher only. The trainer keeps training, and the same command (or sky ai train resume <run>) attaches to it again.

From code

import Std.Db as Db exposing (Db)
import Std.Ai.Dataset as Dataset
import Std.Ai.Train as Train


supportJob : Dataset.Version Dataset.Chat -> Train.Job
supportJob version =
    Train.lora Train.devBase version
        |> Train.withName "support"
        |> Train.withQuantisedBase
        |> Train.withRank 16
        |> Train.withMaxSteps 60
        |> Train.withSeed 7
        |> Train.withGgufExport


trainSupport : Db -> String -> Task Error Train.Adapter
trainSupport db digest =
    Dataset.load (Dataset.folder ".sky/ai/datasets" "support") digest Dataset.chatCodec
        |> Task.andThen (\version -> Train.drive db (supportJob version))

Train.drive starts the run, advances it until it ends and returns the adapter. A program that must not block (a web app, a worker) uses Train.start and Train.poll instead (The durable workflow).

The backend is not in the code. On a Mac with Apple silicon the run uses MLX. On Linux with an NVIDIA GPU it uses Unsloth. In CI and in tests, SKY_AI_TRAIN_BACKEND=fake runs the scripted Fake trainer.

The job

A job is a base model, a frozen dataset version and its settings.

FunctionDatasetThe loss counts
lora base versionDataset.Version Dataset.Chat ({"messages": [...]})the last assistant turn of each conversation
loraOnInstructions base versionDataset.Version Dataset.Instruction ({"prompt", "completion"})the completion

A job takes a frozen Version, never a live dataset. So the type fixes the lineage: the adapter records the version's name and digest.

BuilderDefaultMeaning
withName nadapterthe run directory's prefix and the registry model name. Lower case letters, digits and -, at most 48 characters, not starting with -
withQuantisedBaseoffmark the job QLoRA (the base weights are 4-bit). The same as withQuantBits 4. Recorded in the lineage and the registry method
withQuantBits b16the bits the base is held in while training: 4 or 8 quantise a 16-bit base as it loads, 16 keeps it. An already-quantised snapshot trains as it is. SKY_AI_TRAIN_QUANT_BITS
withMemoryCap bytesthis machine's memory (three quarters of it on Apple silicon)the most memory the trainer may use. A job whose estimate is above it is ModelTooLarge before it starts, and the native trainer's guard ends a run that goes over it with ModelTooLarge, never relaunching it. SKY_AI_TRAIN_MEMORY_CAP (bytes, or 2500MB, 2.5GB, 2GiB)
withRank r8the LoRA rank. SKY_AI_TRAIN_RANK
withAlpha atwice the rankthe update is scaled by alpha / rank
withLayers n16how many of the last decoder layers get adapters. -1 for all
withTargets tsthe four attention projectionsthe projections that get adapters, as Hugging Face module paths
withLearningRate r1e-4the learning rate (Adam)
withEpochs n1passes over the train split. withMaxSteps wins
withMaxSteps n0 (not set)stop after this many optimiser steps
withBatch { perDevice, accumulate }{ perDevice = 4, accumulate = 1 }rows per step, and the gradients accumulated over that many of them. SKY_AI_TRAIN_BATCH (4, or rows x accumulate: 1x4)
withSequenceLength n1024tokens per row. A longer row is cut. SKY_AI_TRAIN_SEQUENCE_LENGTH
withSeed s0the seed of the adapter's initial values and of the data order
withCheckpointEvery n100write a checkpoint every n steps (0: none)
withEvalEvery n50measure the eval split's loss every n steps (0: at the end only)
withBackend bdetectedtrain on this backend unless SKY_AI_TRAIN_BACKEND says otherwise
withGgufExportoffalso convert the adapter to GGUF
withMergedExportoffalso write a merged GGUF model. It implies withGgufExport
withWorkDir d.sky/ai/runsthe directory that holds the run directories
withRegistry rnoneregister the finished adapter in this registry
withMaxRelaunches n3how often a dead trainer is relaunched before the run fails
withWatchWindow ms300000the longest one watch of the trainer lasts (at most five minutes)
fakeCrashAt n0Fake only: the first attempt dies after step n
fakeStepMs n20Fake only: the time one step takes

An operator overrides the six settings marked with a variable from the environment, and the variable wins over the code. Train.effective job is the job as it runs here, and Train.plan lists each setting with its source (env, code or default). A variable that changes the adapter (sequence length, batch, rank, bits) makes a different run, with its own run id. A value that does not parse is InvalidSetting naming the variable and the form it takes; the value itself is never shown.

The targets are self_attn.q_proj, self_attn.k_proj, self_attn.v_proj, self_attn.o_proj (the default) and mlp.gate_proj, mlp.up_proj, mlp.down_proj.

Train.check job checks a job without I/O. A setting out of range is InvalidSetting with the setting's name: a bad name, a rank below 1, a negative alpha, withLayers 0 or below -1, no target or an unknown one, a learning rate of 0 or less, no epoch and no step, a batch of no rows, a sequence below 16 tokens, a negative checkpoint or eval interval, or a negative relaunch count. An empty train split or an empty eval split is DatasetInvalid, because the eval split measures the adapter.

Train.settings job returns every setting. Train.totalSteps job is the optimiser steps the job runs: maxSteps when set, else whole epochs of the train split, rounded up.

The run's identity

Train.jobJson job is the job as canonical JSON with sorted keys. It holds the settings that decide the adapter: the base, the serving model, the dataset name and digest, the format and the training settings. It does not hold the backend (the environment picks it), the work directory, the watch window or the relaunch count.

Train.runId job is the job's name and the first twelve hex digits of the SHA-256 of that JSON, for example support-3f9a1c0b2d4e. Train.runDir job is <workDir>/<runId>. The same job is the same run, so starting it again attaches to it and does not train twice. A changed setting is a new run.

The backend is not part of the identity. A job that finished on Fake is the same run on Mlx. To train the same job on another backend, give it another name or another work directory.

Base models

A BaseModel is an id, the pinned weights each backend trains on, and the GGUF model that serves the adapter.

myBase : Train.BaseModel
myBase =
    Train.baseModel "Qwen/Qwen2.5-0.5B-Instruct@7ae557604adf67be50417f59c2c2f167def9a775" Local.devModel
        |> Train.withWeights Train.Mlx
            { repo = "mlx-community/Qwen2.5-0.5B-Instruct-4bit"
            , revision = "a5339a4131f135d0fdc6a5c8b5bbed2753bbe0f3"
            , files = [ { path = "config.json", sha256 = "b045e57e…" }, … ]
            }

Train.devBase is the development base: Qwen2.5 0.5B Instruct (Apache-2.0). Mlx trains on the mlx-community 4-bit snapshot (QLoRA, about 280 MB), Unsloth on the unsloth bnb 4-bit snapshot, and the adapter is served on Local.devModel, the Q4_0 GGUF of the same model.

Any other base can come from a reference, with the model that serves it:

Train.baseFromReference
    "hf:Qwen/Qwen2.5-1.5B-Instruct@989aa7980e4cf806f80c7fef2b1adb7bc71aa306"
    servingModel
    |> Task.andThen Train.pinWeights

baseId, baseServing and baseWeights backend base read a base. Train.provisionBase backend base downloads and verifies a backend's weights and returns their directory ("" for Fake). A base without weights for the backend is InvalidSetting.

Backends

type Backend
    = Mlx       -- the native trainer (sky-trainer, MLX) on Apple silicon
    | Unsloth   -- Unsloth core on Linux x86_64 with an NVIDIA GPU (CUDA 13)
    | Vertex    -- the same driver in a container, as a Vertex AI custom job
    | Fake      -- the built-in scripted trainer: tests, CI, a machine with no accelerator

The backend is resolved in this order:

  1. SKY_AI_TRAIN_BACKEND (mlx, unsloth, vertex or fake), set by the operator. Under a project prefix the name is <PREFIX>_AI_TRAIN_BACKEND.
  2. Train.withBackend in code.
  3. Detection: Mlx on macOS arm64, Unsloth on Linux with an NVIDIA driver. Elsewhere the run fails with NoAccelerator, and the message names SKY_AI_TRAIN_BACKEND=fake.

Any other value of the variable is InvalidSetting naming the variable. sagemaker is InvalidSetting too: the AWS backend is not supported yet. vertex is never detected: a cloud job is chosen in code or by the variable.

A backend this host cannot run is NoAccelerator, never a silent fallback to another one. Mlx needs Apple silicon. Unsloth needs Linux x86_64 with an NVIDIA GPU and a driver for CUDA 13 (the PyPI torch wheels need driver 580 or newer). Fake runs everywhere a Sky program runs: it is the program itself.

SKY_AI_ACCEL=cpu runs MLX on the CPU (a CI runner without Metal).

Mlx trains with sky-trainer, Sky's native LoRA trainer (C on MLX, no Python). sky ai provision --train mlx installs the pinned bundle for the host. Where no bundle is pinned yet, build one with scripts/ai/trainer/build.sh and set SKY_AI_TRAINER_BIN to it.

The bundles are built in CI only, by .github/workflows/ai-trainer-bundles.yml (scripts/ai/trainer/build-trainer-bundle.sh), from MLX 0.32.4, mlx-c 57731b00e and their dependencies, each pinned by SHA-256. There are two: darwin-arm64 (Metal, with the kernels in lib/mlx.metallib) and linux-x64 (the CPU, on OpenBLAS linked into libmlx.so, built with a pinned GCC 13 (scripts/ai/trainer/gcc13-debs.txt) whose C++ runtime, libstdc++ and libgcc_s, ships in lib/; it needs glibc 2.35 or newer and nothing else). Each carries sky-trainer, lib/, the licences, a NOTICE.txt and an SPDX SBOM. The unpacked archive passes scripts/ai/trainer/test-trainer.sh on the tiny model before the sky-trainer-v<version> release publishes it, and ai_pins.go pins each archive's SHA-256.

Train.plan job returns the resolved backend and where it came from: env, code or detected. Train.backendName gives the name the variable uses.

Every trainer speaks one protocol (the events.jsonl lines), uses one checkpoint format and reads one data order, so a run gives the same files and the same lines on each:

Metal kernels: ahead of time

The darwin-arm64 bundle compiles MLX's Metal kernels ahead of time (MLX_METAL_JIT=OFF), into lib/mlx.metallib. The workflow also builds an MLX_METAL_JIT=ON bundle on every branch push, as a measurement that is never published. Measured (run 40333010053, the selftest on an Apple M-series GPU):

ArchiveUnpackedFirst launchSecond launch
ahead of time (shipped)43.6 MB161 MB4.2 s2.0 s
MLX_METAL_JIT=ON6.2 MB26 MB8.9 s2.1 s

The JIT build compiles each kernel the first time a process uses it; the macOS shader cache then keeps it, so the cost is about 5 seconds once per machine. The bundle stays ahead of time because the download is once per trainer version and smaller than the base model it trains, while a kernel that fails to compile fails at build time in CI, never in a user's first training step.

The trainers

BackendTrainerHostInstalled from
mlxsky-trainer (native)macOS arm64the pinned bundle sky-trainer-<version>-darwin-arm64.tar.gz
fakethe runtime itselfeverywherenothing to install (fake@builtin-1)
unsloththe Python driverLinux x86_64, CUDA 13the managed Python toolchain (below)

Install a trainer ahead of time:

sky ai provision --train mlx              # or unsloth; fake installs nothing
sky ai provision --train mlx --base dev   # also the dev base's weights

or from code with Train.provision Mlx (and Train.provisionBase Mlx Train.devBase). Both are idempotent: an installed trainer returns at once. Train.toolchainStatus () lists every backend's trainer on this host, installed or not, with its id and its versions.

The native trainer

A bundle holds sky-trainer, lib/ (libmlx, libmlxc, and on macOS mlx.metallib; on Linux the C++ runtime it was built with), the licences, a NOTICE.txt and an SPDX SBOM. It is pinned per host by its file and SHA-256 in runtime-go/rt/ai_pins.go, downloaded, checked (a different digest is ChecksumMismatch and installs nothing), run once with --version, and installed under trainers/mlx/<version>-<variant>/ with ok.json last. A linux-x64 CPU bundle is built and tested with it; the Mlx backend itself runs on Apple silicon only, and CI runs the trainer's own contract (scripts/ai/trainer/test-trainer.sh) with the Linux bundle. An operator who builds their own (on a Mac, scripts/ai/trainer/build.sh) sets SKY_AI_TRAINER_BIN to it; a value that is not a sky-trainer is an error, never a silent fallback to the pinned bundle.

The managed Python toolchain (Unsloth)

The Unsloth toolchain is a pinned uv, a pinned CPython and a virtual environment installed from a hash lock (unsloth-linux-x86_64-cuda13.txt: unsloth 2026.9.11, torch 2.12.1). The system Python is never used. The interpreter is CPython 3.12.14 (python-build-standalone 20260924, install_only_stripped) and the installer is uv 0.12.16. Each archive is pinned by SHA-256 in runtime-go/rt/ai_pins.go.

Layout

Everything goes under the AI home (SKY_AI_HOME, else $SKY_HOME/ai, else the user cache directory, for example ~/Library/Caches/sky/ai on macOS):

trainers/mlx/<version>-<variant>/   the native trainer bundle (ok.json last)
bases/<org>--<repo>@<revision>/     a pinned base snapshot
tools/uv/<version>/uv               (Unsloth only)
python/<tag>-<target>/bin/python3   (Unsloth only)
toolchains/unsloth/<id>/venv/       the locked packages
toolchains/unsloth/<id>/driver/     the Sky training driver
toolchains/unsloth/<id>/sbom.json   every package with its version and hashes
toolchains/unsloth/<id>/ok.json     written last: the toolchain is complete
cache/uv/                           uv's wheel cache

The toolchain id is the first 16 hex digits of a SHA-256 over every pinned input: the lock, the CPython archive, the uv archive and the driver's files. A changed pin or a changed driver installs a new toolchain beside the old one. It never changes an installed one.

How the Python toolchain is installed

  1. uv and CPython are downloaded, checked against their digests, unpacked in a staging directory, run once to check their version, and renamed into place.
  2. uv venv --relocatable makes the environment in a staging directory.
  3. uv pip sync --require-hashes --only-binary :all: --no-config installs the lock from PyPI. No wheel installs unless its hash is in the lock, and no source build runs. The index is trusted for availability only, never for content.
  4. The installed versions are read back from the environment's metadata and must equal the lock.
  5. The driver is copied in and runs its self-check, which must print a hello line with the versions it carries.
  6. sbom.json and then ok.json are written, and the directory is renamed into place.

The install runs under a lock file, so two programs that provision at once take turns, and the second finds the first's toolchain. uv runs with a cleared environment: the engine's allow-list (PATH, HOME, TMPDIR, the locale, the library paths, the GPU device selectors) plus the proxy and certificate variables. A PIP_* or UV_* setting of yours, a token or a key never reaches it.

scripts/ai/lock-toolchains.sh regenerates the lock with uv pip compile --generate-hashes --only-binary :all: for its platform and Python 3.12, with a fixed publication cut-off, so running it again gives the same lock. A lock change is a reviewed edit.

Fetching

With fetching allowed (SKY_AI_FETCH=on, the default in development), the workflow installs a missing trainer and downloads a missing base. With fetching off (the default in production), a missing trainer or base is NotProvisioned, and the failure names the command to run: sky ai provision --train <backend>, or sky ai provision --train mlx --base dev for the weights. Provision a production host or an image ahead of time.

Plan before you train

Train.plan job : Task Error Train.Plan

plan checks the job, resolves the backend, reads the host and the serving model's shape (fetching the model when allowed) and estimates the memory. It returns the run id, the backend and its source, the host, the shape, the steps, the row counts, the estimate and the memory the backend may use. A job that will not fit is ModelTooLarge before anything is downloaded or started. A backend this host cannot run is NoAccelerator. Fake is never too large.

The workflow runs the same check as its plan step, after the toolchain and the base weights are in place and before the trainer starts: a job that will not fit ends as ModelTooLarge without launching a trainer. Call plan yourself to learn this before anything is downloaded.

Train.estimateMemory shape job is the native trainer's estimate, in bytes. It is a documented formula, deliberately on the high side, calibrated against the trainer's measured peak footprint on Metal (Qwen2.5 0.5B 4-bit, rank 8, 8 layers, q and v): 0.71 GB for one row of 256 tokens, 0.72 GB for 1 × 512, 0.83 GB for 1 × 1024 and 1.75 GB for 4 × 1024, where the formula gives 0.79, 0.86, 1.09 and 2.10 GB. The trainer computes the loss over chunks of tokens and backpropagates one layer at a time, so only one chunk's logits and one layer's activations are live at once:

A Python backend (Unsloth) keeps its own estimate: its activations for every adapted layer, 8 bytes per logit and 512 MiB for the Python runtime.

Rows is perDevice. The memory the backend may use is three quarters of the total memory on Apple silicon (Metal works within about that much of the shared memory), and the memory available now elsewhere. Train.host () reads the machine (accelerator is metal, cuda or cpu), and Train.shapeOf base reads the serving model's shape.

The durable workflow

Train.workflow name job is a Std.Durable workflow. Each line is a journalled step:

StepWhat it does
resolvethe backend (env > code > detection)
stagethe version's Train and Eval rows into the run directory
toolchainthe managed toolchain, installed if allowed
basethe backend's base weights, downloaded and verified
shapethe serving model's shape (fetched if allowed)
originthe run directory's last attempt, if any
launch-<a>the trainer, detached (attaches when it already runs)
watch-<a>-<n>, tick-<a>-<n>watch up to the window, then a durable one-second sleep
resume-<a>, launch-<a>a dead trainer relaunched from its newest checkpoint
collectthe PEFT adapter's digest, checked against the trainer's
convertthe GGUF adapter (withGgufExport)
mergethe merged GGUF model (withMergedExport)
registerthe registry version (withRegistry)

The run ends with its result in <run>/result.json: the adapter, or the typed failure. progress, result and drive read it.

FunctionUse
start db jobcreate the run and return its id. Idempotent: a run that exists is left alone. It checks the job and resolves the backend first, so a bad setting or a backend this host lacks fails at once and starts no run
poll db jobsadvance the runs of these jobs that are due (one worker tick). True when it did work
drive db jobtrain in the foreground: start, then advance until the run ends, and return the adapter or the run's typed failure. A finished run returns its adapter at once
workflows jobsthe workflows of several jobs, for a worker's own Durable.poll list
workflow name jobthe workflow under your own name

drive needs a database for the durable journal (Std.Db, SQLite or PostgreSQL). sky ai train uses .sky/ai/train.db (SQLite) unless --db names another.

The trainer runs detached

The trainer is not a Process.spawn child, because such a child dies with the program, and a training run must outlive the program that started it. It starts in a session of its own (on Unix), with its standard output going to <run>/events.jsonl and its standard error to <run>/stderr.log. The workflow reads the events file from an offset. So:

Watching, and the lease

Each watch is a journalled step that waits up to the watch window for the attempt to end (a terminal protocol line, or the trainer gone). Between two watches, a one-second durable sleep releases the run. A watch lasts at most five minutes, so every durable step ends well inside the 30-minute lease, and a long run never holds a worker past it. withWatchWindow sets a shorter window (a web app that polls each second uses 2000).

Relaunch from the newest checkpoint

When a watch finds the trainer gone without a terminal line (a SIGKILL, an out-of-memory kill, a power cut), the workflow relaunches it. The next attempt starts from the newest checkpoint whose files match its manifest, so it repeats at most withCheckpointEvery steps. After withMaxRelaunches relaunches the run fails with TrainingFailed. With the default of 3, a run is launched at most four times.

The data order depends only on the seed and the step: each epoch is a seeded permutation of the train rows, and step s reads the rows at positions s × batch to (s + 1) × batch of that order. So a resumed attempt reads exactly the batches the first attempt would have read. The checkpoint holds the optimiser state too, so the resumed attempt continues Adam where it stopped (Measured on a Mac).

A job checks the durable run it advances: a worker whose job differs from the job that started the run refuses it with InvalidSetting (a new job is a new run).

The run directory

<workDir>/<runId>/ is the source of truth for a run. progress, cancel, result and the CLI read it, so they need no database.

job.json              the job as Train.jobJson encodes it
stage.json            the staged version's digest and row counts
data/train.jsonl      the frozen version's Train rows
data/valid.jsonl      its Eval rows
driver.json           the configuration of the attempt the driver runs
trainer.json          pid, launch token, attempt, resumeFrom, start time, toolchain
events.jsonl          the driver's protocol lines (its standard output)
stderr.log            the driver's standard error
checkpoints/<step>/   adapter.safetensors, optimizer.safetensors, state.json, manifest.json
adapter/              the PEFT adapter (adapter_model.safetensors, adapter_config.json)
adapter.gguf          the GGUF adapter (withGgufExport)
merged.gguf           the merged model (withMergedExport)
cancel                present once a cancel was asked for
generation            how often the run was resumed
result.json           the finished run's record
cli.json              the sky ai train arguments that made the run (CLI runs only)

Staging hashes the version's data file first. A file whose bytes changed after it was frozen is ChecksumMismatch. Each staged row is the row's own JSON (the chat or instruction format). Staging the same version again writes nothing.

Checkpoints

A checkpoint is checkpoints/<step>/, the step written with seven digits (checkpoints/0000040/):

FileHolds
adapter.safetensorsthe adapter's weights
optimizer.safetensorsthe optimiser state
state.jsonthe step, the loss, the attempt
manifest.jsonthe step and the SHA-256 of every other file

The driver writes a checkpoint into a temporary directory, flushes every file to disk, writes the manifest last and renames the directory into place. Only then does it write the checkpoint line. A reader that sees the line sees a complete checkpoint.

Before a resume, the runtime hashes every checkpoint's files against its manifest. A checkpoint that does not match (a torn write, a disk error) is skipped, and the resume falls back to the one before it. The driver checks the manifest again before it loads the checkpoint.

Checkpoints hold safetensors and JSON only, never a pickle.

Cancel and resume

Train.cancel job : Task Error Bool
Train.resume job : Task Error Int

cancel writes the cancel file, then sends SIGTERM to the trainer's process group (only when the process is still the trainer). The driver stops at the next step boundary, writes a checkpoint and a cancelled line, and exits. A trainer that is still running after 60 seconds gets SIGKILL. cancel returns True when it signalled a live trainer. The run ends with TrainingCancelled the next time it is advanced (poll or drive). A cancel of a directory with no run is NotRegistered.

Off Unix there is no SIGTERM: the driver reads the cancel file at every step, and the kill after the grace period is the hard stop.

resume starts a failed or cancelled run again. It counts a new generation, removes result.json and the cancel file, and returns the generation. Each generation is its own durable run, so the next start or drive takes the run up and relaunches the trainer from its newest verified checkpoint. A run whose trainer still runs is not resumed (InvalidSetting: cancel it first, or wait).

cancelAt dir, resumeAt dir, progressAt dir and exportAt dir merged do the same for a run directory, without the job. The CLI uses them.

Progress

Train.progress job : Task Error Train.Progress
Train.result job : Task Error (Maybe Train.Adapter)
Train.runs workDir : Task Error (List Train.Progress)

progress folds the run's protocol lines into a Progress: the status, the backend, the step of the total, the latest loss and eval loss, the tokens per second, the attempt, the relaunches and the step the last one resumed from, the last checkpoint, the loss curves (losses and valLosses, at most 200 points each, thinned evenly) and a message. A line it cannot decode is noted in the message and skipped, so progress never fails on one bad line.

The status is one of Queued (no run directory yet), Provisioning (the data is staged and the toolchain, the base and the plan are being prepared, before the first trainer starts), Running, Relaunching (the trainer is gone and no result is written yet), Converting (the trainer finished and the collect and convert steps run), Registering (the GGUF adapter is written and the merge and register steps run), Done, Failed and Cancelled. statusText gives the lower-case name.

result is Nothing while the run has not finished, the adapter when it has, and the run's typed failure when it failed. runs workDir lists every run in a work directory, newest first.

Status(..) exposes Done, and Sky.Core.Task exposes Step(..) with a Done too. Do not expose both unqualified in one module.

In the Sky Console

The console's AI tab has a Training section. It lists the runs this process touched and the runs under .sky/ai/runs, each with its status, its step of the total, its losses, its relaunches, its last checkpoint, the version it registered, the error of a failed or cancelled run, and a small loss curve. A row holds no dataset text, no prompt and no secret (../observability.md).

The driver protocol

The trainer writes one JSON object per line to events.jsonl. Every line has "v": 1 and a type. The driver writes the keys sorted, and writes a non-finite number as null (which Sky reads as -1).

typeFieldsMeaning
hellobackend, toolchain, versions, attempt, resumeFrom, totalSteps, pidan attempt starts
progressstep, total, loss, learningRate, tokensPerSecond, epochone optimiser step is done
metricstep, name, valuea measurement. name is val_loss for the eval split's loss
checkpointstep, patha complete checkpoint is on disk
artefactkind, path, sha256a file the run produced. kind is peft for the adapter
cancelledstepthe attempt stopped after a cancel (terminal)
donestep, trainLoss, valLossthe attempt finished (terminal)
errorcode, messagethe attempt failed (terminal)

The code of an error line is a Std.Ai.Failure name: DatasetInvalid, NoAccelerator, ToolchainFailed, ModelTooLarge (the trainer ran out of memory) or TrainingFailed. Any other code is read as TrainingFailed.

Train.decodeEvent line decodes one line into an Event (Hello, StepProgress, Metric, Checkpoint, Artefact, DoneEvent, CancelledEvent, DriverError). A line of a type this release does not know is Ok Nothing and is skipped, so a newer driver may add types. A line that is not protocol JSON, a line without its type's fields, and a line whose v is not 1 are ToolchainFailed (the toolchain and the runtime differ). Train.foldProgress event progress folds one event into a Progress.

A watch reads complete lines only. A line still being written (no newline yet) is left for the next watch.

The driver claims standard output for protocol lines. Anything else a library prints to standard output (a print, a C extension's message) goes to standard error, so the events file holds protocol lines only.

GGUF and merged export

The training output is a PEFT adapter (adapter/adapter_model.safetensors and adapter_config.json). The managed llama-server loads GGUF, so withGgufExport converts it to adapter.gguf.

The converter is Go, in the runtime. It needs no Python and no PyTorch. It writes the layout llama.cpp's own LoRA converter writes: one pair of F32 matrices for each adapted projection (blk.<n>.<tensor>.weight.lora_a and .lora_b), F32 because an adapter is small and F32 loses nothing. The header has these keys:

KeyValue
general.architecturethe serving model's (qwen2, llama, …)
general.typeadapter
adapter.typelora
adapter.lora.alphathe adapter's alpha
sky.adapter.base_idthe base the adapter was trained on (<org>/<repo>@<commit>)
sky.adapter.serving_sha256the serving GGUF's digest

Every matrix is checked against the serving model's shape. An adapter that does not fit (another layer count, another projection size) is AdapterMismatch before an engine sees it. The written file is read back and checked. Converting the same adapter again gives the same bytes.

withMergedExport also folds the adapter into the serving model's weights with llama-export-lora from the pinned engine build and writes merged.gguf, a model an engine serves with no adapter. With your own engine build (SKY_AI_LLAMA_BIN), llama-export-lora must be next to llama-server, else the merge is NotProvisioned. The adapter stays the product: the registry holds adapters only.

Train.exportAt dir merged converts the finished adapter in a run directory, whether or not the job asked for it, and with merged also writes the merged model. The serving model and the base come from the run's job.json. sky ai export <run> calls it.

Registry and serving

withRegistry reg adds the register step. It calls Registry.register with the job's name, the GGUF adapter when there is one (else the PEFT safetensors), and the adapter's lineage:

The method is Qlora for a job with withQuantisedBase, else Lora. Train.registryAdapter adapter and Train.lineage adapter build the same values for your own Registry.register call. The adapter's registered field holds the label (support@v1). Registering the same bytes again returns the same version, so a resumed workflow does not create a second one. Registry.trainedOn reg digest then finds every version trained on a dataset.

Serve a registered adapter through a registry alias (registry.md), or serve the file directly with Local.withAdapter:

supportSpec : Train.Adapter -> Local.Spec
supportSpec adapter =
    case Train.adapterRef adapter of
        Just ref ->
            Local.model Local.devModel |> Local.withAdapter ref

        Nothing ->
            Local.model Local.devModel

Train.adapterRef is the GGUF adapter as a pinned file (Local.file { path, sha256 }), Nothing for a job without withGgufExport. The engine hashes the file before each load. An adapter that Std.Ai.Train converted records the model it was converted for, and loading it on another model is AdapterMismatch. sky ai start <slot> --adapter file:<path>#sha256=<hex> does the same from the command line.

The adapter record

drive and result return an Adapter:

FieldHolds
name, runIdthe job's name and the run
baseId, servingthe base's id and the serving model reference
dataset, datasetDigestthe frozen version
backend, toolchainthe backend, and <name>@<id> of the toolchain
methodlora or qlora
seed, stepsthe seed and the steps trained
peftthe PEFT adapter's path and SHA-256
gguf, mergedthe GGUF adapter and the merged model, when asked for
trainLoss, valLossthe last step's loss and the eval split's loss
relaunches, resumedFromthe relaunches and the checkpoint step the last one resumed from
registeredthe registry label, with withRegistry
jobthe job's JSON

Train.adapterCodec is its JSON form (the adapter field of result.json).

Tests

Under sky test in test mode (a project that commits a .env.test, ../tooling/testing.md) no trainer process starts and no network is used. The durable workflow runs for real on the test database, and the launch step writes what a Fake trainer writes: the protocol lines, verified checkpoints and a real, no-op PEFT adapter. The toolchain, the base and the model shape need no download. The GGUF conversion runs. The merge writes a model-shaped stand-in. So a program's own tests run the whole workflow, including the registry, in milliseconds.

Put SKY_AI_TRAIN_BACKEND=fake in .env.test, so the test does not depend on the host (on a Linux runner without a GPU, detection is NoAccelerator).

fakeCrashAt n makes the first attempt stop after step n with no terminal line, as a killed trainer does, so a test sees the relaunch from the newest checkpoint:

crashy : Train.Job -> Train.Job
crashy job =
    job
        |> Train.withMaxSteps 12
        |> Train.withCheckpointEvery 2
        |> Train.fakeCrashAt 3

A run of crashy job finishes at step 12 with relaunches = 1 and resumedFrom = 2 (the newest checkpoint before step 3). examples/73-ai-train/tests/TrainExampleTest.sky asserts this, then serves the registered adapter through a slot answered by a fixture under tests/mocks/.

Outside test mode, Fake runs a real trainer process: the program starts itself again as the trainer (--sky-ai-fake-trainer), so the launch, the watch, a crash, the relaunch and cancel are all real. Nothing is installed for it. It needs the serving model for its shape.

Failures

Every failure is an Error with a typed Std.Ai.Failure (local.md). Read it with Failure.fromError, and keep a _ -> branch:

explain : Error -> String
explain err =
    case Failure.fromError err of
        Just (TrainingCancelled c) ->
            "cancelled at step " ++ String.fromInt c.step

        Just (TrainingFailed f) ->
            "failed at step " ++ String.fromInt f.step ++ ": " ++ f.reason

        Just (NotProvisioned p) ->
            "run " ++ p.command

        _ ->
            Error.toString err
FailureWhenErrorKind
TrainingFailed { runId, step, reason, logTail }the trainer reported an error, or died more often than withMaxRelaunches allows, or a run file cannot be writtenUnavailable (not retryable)
TrainingCancelled { runId, step }the run was cancelledConflict
ModelTooLargeplan finds the job will not fit, or the trainer ran out of memoryInvalidInput
NoAcceleratorthe backend cannot run on this hostUnavailable
DatasetInvalidan empty train or eval split, a data file that is not frozen rowsInvalidInput
NotProvisioneda toolchain, a base or llama-export-lora is missing and fetching is off. It names the commandNotFound
ToolchainFailedan install step, the self-check, the trainer's start, a protocol line, or llama-export-lora failedUnavailable
ChecksumMismatcha dataset file changed after it was frozen, a download, or the adapter's digest differs from the trainer'sConflict
AdapterMismatchthe adapter does not fit the serving model, or an engine runs another modelInvalidInput
InvalidSettinga setting out of range, a bad SKY_AI_TRAIN_BACKEND, vertex or sagemaker, a resume of a running trainer, a worker whose job differsInvalidInput
NotRegistereda cancel or resume of a directory with no runNotFound

Security

The CLI

sky ai provision --train mlx|unsloth|fake [--base <base>]
sky ai train --data <dir>/<name>[@<digest>] [--base <base> --serving <model>]
             [--name <n>]
             [--steps <n>] [--rank <r>] [--seed <s>] [--layers <n>]
             [--checkpoint-every <n>] [--eval-every <n>]
             [--backend mlx|unsloth|fake] [--gguf] [--merged] [--register]
             [--detach] [--work-dir <dir>] [--db <dsn>]
sky ai train status [<run>]
sky ai train runs
sky ai train resume <run>
sky ai train cancel <run>
sky ai export <run> [--out <file>] [--merged]
sky ai start <slot> --adapter file:<path>#sha256=<hex>

Every verb takes --format json. The verbs run the bundled Sky program (sky-bundled/ai-train, built the first time a training verb runs, apart from the engine and registry program sky-bundled/ai), which calls the API above.

Measured on a Mac

On an Apple M-series Mac with 16 GB, the native trainer (sky-trainer) on the 4-bit development base (rank 8 on the last 8 layers, q and v, Metal) peaked at these memory footprints, with these throughputs:

rows × tokenspeak footprinttokens per second
1 × 2560.71 GB689
1 × 5120.72 GB706
1 × 10240.83 GB725
4 × 10241.75 GB741

The loss is computed over chunks of tokens and the backward pass runs one layer at a time, so the logits of the whole batch never exist at once. Before that, 1 × 256 peaked at 1.77 GB and each token added about 5.3 MB.

The same trainer driven from Sky (ai_train_native_live.rs: 24 steps of 1 × 256 under a 2.5 GB cap, the trainer's process group killed by SIGKILL after its step-8 checkpoint) relaunched once from step 8, finished at step 24 with an eval loss of 2.89 and a peak footprint of 0.61 GB, and its GGUF adapter answered a chat on the managed engine after the trainer had exited. The whole test took 25 s.

The Go converter's output loads in the pinned llama-server b11200 (its /lora-adapters route lists it), and llama-export-lora merges it into the Q4_0 dev model.

The live tests are rust/crates/sky/tests/ai_train_live.rs (the Fake trainer as a real process of the app, a crash and a relaunch, the real engine; and the native trainer's contract, scripts/ai/trainer/test-trainer.sh, on the tiny model with the trainer a user gets), ai_train_native_live.rs (a QLoRA run with the native trainer on Apple silicon under a 2.5 GB cap, whose trainer is killed after its first checkpoint and resumed) and runtime-go/rt/ai_python_suites_live_test.go (the Unsloth driver's Python suites under the pinned CPython, go test -tags ailive). Without their trainer or engine they fail and name what to install, unless SKY_LIVE_TESTS=skip. CI takes the trainer from the pinned bundle, or until a bundle is pinned from the newest ai-trainer-bundles.yml run (.github/actions/sky-trainer).

Training on Vertex AI

The Vertex backend runs the same driver in a container, as a Vertex AI custom job on Google Cloud, and returns the same adapter a local run gives. On a machine with GPUs (g2-standard-4 carries one NVIDIA L4) the container runs the Unsloth driver; on a CPU machine (n2-standard-4) it runs the Fake driver, which tests the whole path for cents.

supportJob : Train.Job
supportJob =
    Train.lora Train.devBase version
        |> Train.withQuantisedBase
        |> Train.withBackend Train.Vertex
        |> Train.withCloud
               { project = "", region = "us-central1", machine = "g2-standard-4", staging = "" }
        |> Train.withBudget (Money.fromMajor Money.USD 2)
        |> Train.withMachinePrices prices
        |> Train.withMaxRuntimeMinutes 40
BuilderVariableNotes
withCloudSKY_AI_TRAIN_PROJECT, _REGION, _MACHINE, _STAGINGthe staging location is gs://<bucket>/<prefix>, in the job's region; machine default g2-standard-4
withBudgetSKY_AI_TRAIN_BUDGET (USD 2.00)required
withImageSKY_AI_TRAIN_IMAGEthe trainer image, pinned by digest (sky ai image build --train)
withServiceAccountSKY_AI_TRAIN_SERVICE_ACCOUNTwhat the job runs as
withLabelsSKY_AI_TRAIN_LABELS (team=support,env=qa)added to Sky's own labels
withMachinePricesnone (code only)machinePrices currency version rates
withMaxRuntimeMinutesnonea ceiling below the budget's
withSpotnoneSpot capacity: a pre-empted attempt is relaunched
withKeepStagingnonekeep the run's objects and job records

The image and the one-time project setup are in deploy-gcp.md. Train.cloudSettings resolves and checks the settings without touching the cloud; Train.start runs that check first, so a job with no project, no budget or an image without a digest starts no run.

The price book and the budget

The stdlib ships no prices. A job states the ones it trusts:

prices : Train.MachinePriceBook
prices =
    Train.machinePrices Money.USD "2026-10-12"
        [ { machine = "g2-standard-4", region = "us-central1", perHour = "0.95" } ]

A rate is the machine's whole hourly price (cores, memory, its GPUs, the management fee and the boot disk). sky ai cloud prices reads the Cloud Billing catalogue (see deploy-gcp.md). A machine and region that are not in the book are InvalidSetting: with no price there is no estimate.

Train.plan (and the workflow's cloud-plan step) estimates the job, pure: 15 minutes to provision and pull the image, the steps, and 5 minutes to export and upload, at the book's rate. A step is the job's fakeStepMs plus 50 ms on a CPU machine, and on an L4 the larger of 250 ms and parameters / 10⁹ × tokens per step / 4000 ms (Train.cloudEstimate). An estimate above the budget is BudgetExceeded, and nothing launches. The plan also reads the staging bucket's location (a bucket outside the job's region, or a multi-region one, is InvalidSetting: the data stays in the region) and the L4 training quota through Cloud Quotas (QuotaExceeded when it is below the machine's GPUs).

The budget bounds the run three ways:

  1. The job is created with a hard runtime ceiling, scheduling.timeout, the smaller of what the budget buys at the rate and withMaxRuntimeMinutes. Vertex stops the job even when no Sky process is left.
  2. Each watch measures the spend: the time each attempt has run (from its start, or from its creation while it waits), at the rate, plus the earlier attempts'. At the budget the watch cancels the job and the run ends BudgetExceeded.
  3. A relaunch needs at least its provisioning allowance of budget left, and gets only what is left as its ceiling.

The workflow

resolve, cloud (the settings), stage (the frozen rows into the run directory), shape, cloud-plan, upload, origin, resume-<a>, launch-<a>, watch-<a>-<n> with tick-<a>-<n>, download, then the steps a local run has: collect, convert, merge, register, and cleanup.

The container's shim (runtime-go/rt/ai_train_assets/driver/sky_cloud.py) downloads the run's inputs and the base (each file checked against its digest), runs the driver, and uploads each checkpoint before the protocol line that names it, the adapter before artefact and done, and every segment with a create-only precondition. It reads the cancel marker each cycle and forwards Vertex's SIGTERM, so the driver checkpoints and stops.

Cancel

Train.cancel job (and sky ai train cancel <run>) writes the run's cancel marker and asks Vertex to cancel the job. The run ends TrainingCancelled.

Offline

Under sky test, SKY_AI_GCP_ENDPOINT=http://sky-gcp.invalid in the project's .env.test sends every Google Cloud call to the runtime's built-in emulator, which runs the Fake protocol into an emulated bucket the way the container does. examples/74-ai-cloud and rust/crates/sky/tests/fixtures/ai-cloud test the whole workflow this way.

In Std.Ai.Trace

A run records itself in Std.Ai.Trace under its id (Train.runId job, the id start returns). Each step the watch reads is a progress row (the step, the total, the loss, the tokens per second, the checkpoint written at it): Trace.trainingSteps db runId. When the run ends, finished or failed, it records one Training cost row. A Vertex run's row is its measured spend in exact money, so Trace.costs db adds training to the calls and the evals of the lifecycle. A run on this machine has no price, so its row is unpriced (no cost, never a zero). Every row is keyed, so a resumed run writes nothing twice. See trace.md.

Not yet

The AWS backend (SageMaker) is not supported yet: sagemaker is InvalidSetting.