On-device models: Std.Ai.Device

(v0.28.0) A model slot can run on the user's own device: in the browser, in the iPhone and Android apps, and in the Mac app. The server keeps the slot and its policy. The client asks Device.provider "<slot>", an ordinary Std.Ai.Provider, and each call runs on the device when the device holds the model and can run it, and on the server's slot otherwise. The app code is the same on every target, and one eval suite measures both places.

Every target runs the same engine: llama.cpp b11200 with Sky's engine core (scripts/ai/device/sky_llama.cpp), so a device loads the same GGUF model and the same GGUF LoRA adapter the server serves (a Std.Ai.Train adapter included), with the same chat template and the same sampler.

The live API is sky doc Std.Ai.Device. The design is ../design/std-ai-lifecycle.md §14 P5, and examples/75-ai-everywhere is the app that uses all of it.

Quick start

The server owns the slot. Endpoint.withDevice gives it a device spec:

module Main exposing (main)

import Sky.Core.Prelude exposing (..)
import Sky.Core.Task as Task
import Std.Log exposing (println)
import Std.Ai.Provider as Provider exposing (Provider)
import Std.Ai.Endpoint as Endpoint exposing (Endpoint)
import Std.Ai.Local as Local
import Std.Ai.Device as Device


notesSlot : Endpoint
notesSlot =
    Endpoint.endpoint "notes"
        |> Endpoint.withLocal (Local.model Local.devModel)
        |> Endpoint.withDevice (Device.spec Device.Auto |> Device.withContext 2048)


assistant : Provider
assistant =
    Device.provider "notes"


main =
    Provider.chat assistant [ Provider.user "Summarise: train to Paris on Monday." ]
        |> Task.andThen (\r -> println r.content)
        |> Task.run

This program runs on the server, so assistant is the notes slot itself. In a Sky.Spa client (web:app, desktop:mac, mobile:ios, mobile:android) the same Device.provider "notes" runs in the client, and the device decides per call. An app with a device slot is not supported yet on desktop:windows and desktop:linux: the build refuses it, because the desktop device engine runs behind the macOS window only in this version (other desktop apps build as before). On Windows and Linux, build web:app and open it in a browser; the slot then runs in the browser engine when the browser can run it, else on the server, and a DeviceOnly slot fails typed (DeviceUnsupported).

The build checks the pairing: a Device.provider "x" needs an Endpoint.endpoint "x" binding with Endpoint.withDevice in a module the program imports, the binding must be top level with no parameters, and both name the slot with a literal (or a top-level String constant). Each of these is a build error, because the server registers the slot when it starts.

Placement

The spec's placement is required. It says where a call may run:

PlacementThe device can run itIt cannot
Device.Autoon the deviceon the server
Device.DeviceOnlyon the devicea typed failure; the server never answers
Device.ServerOnlyon the serveron the server

DeviceOnly has no fallback route on the server at all, so a slot that must keep its prompts off the server cannot leak them to it.

Configuration: code holds the default, the operator overrides it

No setting is in sky.toml. The spec's builders hold the defaults, and the operator overrides them per slot:

BuilderVariableDefault
Device.spec Auto/DeviceOnly/ServerOnly<PREFIX>_AI_<SLOT>_PLACEMENT (auto, device, server)required in code
Device.withModel ref<PREFIX>_AI_<SLOT>_DEVICE_MODELthe slot's local model (Endpoint.withLocal)
Device.withAdapter ref<PREFIX>_AI_<SLOT>_DEVICE_ADAPTERthe slot's adapter
Device.withContext n2048 tokens
Device.withThreads n (at least 1)<PREFIX>_AI_<SLOT>_DEVICE_THREADSthe engine's own
Device.withGpuLayers n (0 = CPU only)<PREFIX>_AI_<SLOT>_DEVICE_GPU_LAYERSthe engine's own: every layer where the backend has a GPU
Device.withMaxDownload bytes1 GiB
Device.withBackends [ Native, WebGpu, WasmCpu ]all three, in that order
Device.withMetered TrueFalse: no download on a metered connection

Engine settings for experts: threads and GPU layers

llama.cpp is the engine on every target. The defaults suit most apps; an expert can set two engine settings per slot. What each engine does with them:

EnginewithThreads nwithGpuLayers n
iOS app (SkyLlama.xcframework)uses n (default: the cores less two, at most 4)offloads n layers to Metal (default: all); the simulator has no GPU
Android app (libskyllama.so)uses n (default: half the cores, at most 2)CPU build: no GPU
Mac app (managed llama-server)-t n (default: llama-server's)-ngl n (default: all on Metal); a CPU build refuses n > 0 with NoAccelerator
Browser, WebGpuone thread (a single-threaded WebAssembly build)offloads n layers to WebGPU (default: all)
Browser, WasmCpuone threadno GPU

withThreads is an upper bound: an engine never runs more threads than it is given, and the browser engines run one. withGpuLayers above 0 is a requirement, not a hint. Each device reports which backends have a GPU, and a backend without one is not used for the slot: the decision takes the next backend of the list, else the server (Auto), or fails with DeviceUnsupported whose reason names Device.withGpuLayers (DeviceOnly). withGpuLayers 0 runs on the CPU everywhere. A value out of range (threads below 1, a negative layer count, an environment value that is not a whole number) is refused with InvalidSetting naming the builder or the variable when the server reads the slot's offer; it is never dropped.

A model is a Std.Ai.Local reference with its SHA-256 (Local.reference Local.devModel). An adapter is registry:<name>@<alias> (the slot needs Endpoint.fromRegistry) or a pinned file reference.

The build options are flags:

FlagWhat it does
sky ai provision --device web|ios|android|all [--from <dir|archive>]install the engines the builds stage (below)
sky build --ai-web-engine webgpu|cpu|both|nonewhich browser engines a client build stages; default both when installed
sky build --ai-embed <slot>a native app (desktop:mac, mobile:ios, mobile:android) carries the slot's model and adapter, so it never downloads them

Consent, status and the download

Nothing downloads on its own. A device that does not hold the model answers from the server (Auto) or fails with NotProvisioned (DeviceOnly) until the app calls Device.prepare, which the app runs after the user agreed:

-- doc-example: skip  (fragment of examples/75-ai-everywhere)
Device.status "notes"      -- Ready, NotDownloaded { bytes }, Downloading, ServerSide reason, Unsupported failure
Device.prepare "notes"     -- download, check, load; then Ready
Device.evict "notes"       -- remove the files and unload

NotDownloaded carries the size to show in the consent card. While prepare runs, status reports Downloading { doneBytes, totalBytes }; poll it with Sub.every 1000. A download is 64 MiB Range requests. Each chunk is checked against the server's chunk list and the whole file against the SHA-256 in the offer, before the file is marked ready. An interrupted browser download resumes from the last whole chunk.

WhereThe device's store
a browserthe origin-private file system (OPFS), with navigator.storage.persist() requested
iOSApplication Support/sky-ai, excluded from backups
Androidthe app's files directory, sky-ai/
macOS<AI home>/device/files (a model already pulled by Std.Ai.Local is used in place)

A browser that refuses storage (a private window) reports no room, so Auto answers from the server and DeviceOnly fails with StorageDenied.

How each call is decided

Device.decide is the whole policy as a pure function of the device's capabilities, the server's offer and the prompt's length. The first rule that applies wins:

  1. A ServerOnly placement (the code's or the operator's), or code that runs on the server: the server.
  2. An offer the server refused (an adapter not cleared for devices): the server for Auto, NotPromotable for DeviceOnly.
  3. The first backend of the offer's list that the device has, that has not failed this session, and that can hold the model and the adapter. None: the server for Auto, DeviceUnsupported for DeviceOnly.
  4. Files not stored need 1.1 times their size free: else the server for Auto, StorageDenied for DeviceOnly.
  5. Files not stored: NeedsDownload (the server answers this call for Auto; DeviceOnly fails with NotProvisioned).
  6. A prompt longer than three quarters of the device's context: the server for Auto, InvalidSetting for DeviceOnly.
  7. Otherwise the device.

Backends

BackendWhereNotes
Nativethe iOS and Android apps, the Mac appllama.cpp in the app: Metal on an iPhone, the CPU on the iOS simulator; the CPU with two threads on Android (16 KB-aligned, uncompressed libskyllama.so for arm64-v8a and x86_64); the managed llama-server with Metal on a Mac
WebGpuChrome, Edgellama.cpp's WebGPU backend in a Web Worker. It needs JavaScript Promise Integration and a GPU with shader-f16, so Safari and Firefox never get it. A WebGPU engine that registers no GPU device fails with DeviceUnsupported; it never runs on the CPU in silence, and the next call uses the next backend
WasmCpuevery browser with OPFSthe CPU build of the same engine in a Web Worker; models up to 1 GiB

The worker and the engines are same-origin files with content-hashed names (sky-ai-worker.<hash>.js, sky-ai-webgpu.<hash>.{mjs,wasm}, sky-ai-cpu.<hash>.{mjs,wasm}); the build writes the page's sky-ai-engine meta that names them. Nothing evaluates a string, so the strict policy (SKY_CSP=strict: script-src 'self' 'wasm-unsafe-eval') runs them with no violation.

Measured on an Apple M-series Mac (16 GB) with the development model (Qwen2.5 0.5B, Q4_0, 430 MB) and a 24-step Std.Ai.Train MLX adapter, asking one short question: Chromium WebGPU (a Metal adapter) answered in 0.45 s, Chromium CPU in 5.5 s, WebKit CPU in 6.1 s. A local download of the model took 7 s in Chromium and 13 s in WebKit. The iOS 27 simulator (the model downloaded from the backend) and the Android 15 arm64 emulator (the model carried in the app with --ai-embed, 28 s from launch to the answer) answered "The capital of France is Paris." on the device. The Mac app ran the synthetic model on its managed engine. The WebGPU answer with the adapter matched the server's word for word. In examples/75-ai-everywhere a note's summary took 10.1 s on the CPU build and 0.8 s on WebGPU, and the self-check (three prompts on the device, scored on the server) took 27.1 s and 2.8 s, with both pass rates at 100%. Streamed (Provider.openStream), the development model's answer to the probe's question arrived in 7 pieces that joined to the same text in Chromium (CPU and WebGPU), in WebKit, from the server, and in the Android emulator with the model carried in the app.

Streaming

Provider.openStream and Provider.readStream (or Provider.chatStream with a callback) work on Device.provider like on any provider, and the decision is the same as for a call. On the device the engine hands over each piece of text as it writes it: the browser worker, the iOS and Android apps step the engine core (sky_llama_chat_begin / sky_llama_chat_next, whole UTF-8 pieces), and the Mac app reads its managed engine's event stream. When the decision is the server, the client reads POST /_sky/ai/device/<slot>/stream. On the server itself (Sky.Live, tests) it is the slot's own stream. A piece read on the device sends nothing to the server.

The server's routes

A server with a device slot answers:

RouteWhat
GET /_sky/ai/device/<slot>the slot's offer (placement, model and adapter with SHA-256, size and URL, context, backends); ETag, revalidated every 30 s, so an alias move reaches devices
GET /_sky/ai/blob/<sha256>an offered file, with Range requests; only the digests a current offer names (never a path)
GET /_sky/ai/blob/<sha256>.chunks.jsonits 64 MiB chunk digests
POST /_sky/ai/device/<slot>/callthe fallback call: same origin, application/json, at most 64 KiB, 30 calls a minute per session; absent for DeviceOnly
POST /_sky/ai/device/<slot>/streamthe streaming fallback call (Provider.openStream in a client the decision sends to the server): the same rules and rate as /call, answered as server-sent events, one piece per event, a failure as a typed event; absent for DeviceOnly

Privacy

Failures

Every failure is a typed Std.Ai.Failure:

FailureWhen
DeviceUnsupported { backend, reason } (Unavailable)no backend can run the model (DeviceOnly), or a WebGPU engine found no GPU device
StorageDenied { needBytes, availableBytes } (PermissionDenied)the device has no room, or the browser keeps no storage (DeviceOnly)
NotProvisioneda DeviceOnly slot is not downloaded yet
NotPromotablethe adapter is not cleared for devices (DeviceOnly)
ChecksumMismatcha downloaded chunk or file does not match its digest
InvalidSettingthe prompt does not fit the device's context (DeviceOnly)
EngineFailedthe engine could not load the model or the adapter

One eval suite on the device and the server

Eval.prompts suite lists every request a run of the suite sends. Device.recordPrompts label provider prompts asks a provider each of them on the device and keeps the answers; send the Recording to the server, and score it there with the same suite:

-- doc-example: skip  (fragment of examples/75-ai-everywhere/src/Suites.sky)
Eval.runAs rec.label db (Device.replay rec) summaries

Device.replay answers each recorded request as recorded, so the device and the server's slot are measured by the same scorers and thresholds. A replay answers by the messages alone, so record a suite with one sample per case.

Engines and their provenance

The engines are built from the pinned llama.cpp source (b11200, the source archive's SHA-256 is in rust/crates/sky/src/ai_device.rs) with emsdk 6.0.12 and Dawn's emdawnwebgpu v20260423.175430 (web), the Android NDK 27.0.12077973 (android) and Xcode (ios), by scripts/ai/device/build-device-bundles.sh. The ai-device-bundles workflow publishes them as the release sky-llama-b11200, and sky ai provision --device <target> installs a published archive after checking its pinned SHA-256. --from <dir|archive> installs a local build of the same script. The bundles carry their licences (NOTICE.txt; llama.cpp and ggml MIT, nlohmann/json MIT, Emscripten MIT and NCSA, Dawn BSD 3-Clause).

Testing

Under sky test a server-side Device.provider is the slot itself, so its calls are answered by tests/mocks/ like any slot. The device flows are covered by tests/Std/Ai/DeviceTest.sky (the decision, the wire values, record and replay), runtime-go/rt/ai_device_*_test.go (the routes, the worker's digests and its refusal to run WebGPU on the CPU, the Mac bridge), scripts/ai-device-e2e.sh (Chromium and WebKit under the strict policy) and rust/crates/sky/tests/ai_device_native_flow.rs (the iOS simulator, the Android emulator and the Mac app; the probe also streams its answer and checks it arrived in more than one piece). Streaming is covered by runtime-go/rt/ai_stream_test.go (the ai-stream-pieces gate) and by scripts/ai/device/test-engine-core.sh, which builds the engine core for the build machine's CPU and checks that the streamed answer equals the one-call answer in whole UTF-8 pieces. go run ./cmd/sky-tiny-gguf <dir> (in runtime-go/) writes a 0.4 MB synthetic Qwen2 model and adapter that every engine loads, so the flows run without the 430 MB model.