LLM agents — Std.Ai
Status: v1.
Std.Aiis the LLM / agentic layer of the stdlib. It is built ONStd.Durable(an agent run is one kind of durable workflow) and on the typed effect and money layers already in the stdlib. The live API issky doc Std.Ai.Provider/Std.Ai.Agent/Std.Ai.Tool/Std.Ai.Policy/Std.Ai.Trace/Std.Ai.Memory.Pg; this doc is the architecture behind it.v0.28.0 adds model slots and a managed local engine (
Std.Ai.Endpoint,Std.Ai.Local) andProvider.chatWith/Provider.embed: see../ai/local.mdand the lifecycle design,std-ai-lifecycle.md. To test the whole lifecycle (offline, on a local engine, and live on Google Cloud), follow../ai/testing-guide.md.
Two principles the design is built around
- The harness is the product, not the model. A model is a swappable component behind a typed boundary. Everything durable, auditable, and safety-bearing lives in Sky code that does not change when the model does.
- The LLM is never the authorisation boundary. A model proposes; a pure,
reviewable policy decides; only an allowed decision runs an effect. This is
Std.Ai.Policy, and it is structural, not advisory.
Std.Ai.Provider — a swappable chat backend
A Provider is a value. The first adapter speaks the OpenAI chat-completions
wire format, which is one format for hosted OpenAI, vLLM, llama.cpp, and most
gateways, so a local model needs no new code:
provider = Provider.openai (Secret.fromEnv "OPENAI_API_KEY") "gpt-4o-mini"
local = Provider.compatible "http://localhost:8080/v1/chat/completions" key "llama-3"
chat returns the assistant text AND a typed Usage (token counts). cost
prices a usage in Std.Money from a versioned PriceBook, exact and never a
float, so per-task and per-tenant roll-ups are honest.
The router
A router is a Provider that picks a concrete backend per call by a pure
function over the messages (task shape, sensitivity, length). It is itself a
Provider, so it drops in anywhere one does, and it composes (a route may return
another router). The route decision is pure, so it can be logged and later learnt
from outcomes:
Provider.router (\messages ->
if sensitive messages then local else Provider.openai key "gpt-4o-mini")
Bring your own backend — custom
The stdlib ships one wire format (OpenAI chat-completions) on purpose. A backend
it does not ship — a different wire format, a proprietary API such as the OpenAI
Responses API, a local process, or a test double — is added by the user, not the
stdlib, through custom:
Provider.custom (\messages ->
Http.request theRequest |> Task.andThen decodeToChatResponse)
The result is an ordinary Provider, so it drops into router, Agent,
Policy, Trace, and cost unchanged. custom wraps a plain
List Message -> Task Error ChatResponse; chatTools on it degrades to a plain
chat that returns no tool calls, so it works with the prompt-based Tool loop.
A provider with its own NATIVE function-calling wire uses customTools, which
also takes the chatTools implementation and is a full peer of the built-in
openai provider. This is the extension point that keeps the foundation
vendor-neutral: provider-specific logic lives in a user module or a Sky package
(for example a sky-openai that models Chat Completions plus the Responses API),
never in the stdlib.
Some native tool wires need state from one response to send the next request.
The OpenAI Responses API continues with previous_response_id, and Gemini needs
each function call's thoughtSignature sent back. ToolCall.continuation : Maybe String carries that state. It is provider-owned and opaque: the provider
sets it in its chatTools result, nativeToolLoop journals it with the call and
hands it back in the Called and Returned turns, and the tool executor never
sees it. The built-in chat-completions backend sets Nothing. Behind a router
a provider can receive a value another backend set, so it checks that it
recognises the value and replays the history when it does not.
Std.Ai.Agent — an agent as a durable workflow
oneShot is one system prompt plus the caller's messages, one journalled model
call, the answer and usage out. toolLoop is the tool-calling loop: each model
turn and each tool call is its own journalled Durable.step. Because the agent
IS a durable workflow, an expensive, non-idempotent model call becomes
exactly-once across a resume — a crashed run that comes back after the call
replays the recorded answer and does not re-bill the model. Between-turn state
(the message list) is rebuilt deterministically from the journalled steps, so
replay is exact.
agent = Agent.toolLoop "assistant" provider "You are terse." [ lookupTool ] 6
Durable.start db agent "chat-42" { messages = [ Provider.user "..." ] }
Durable.poll db [ Durable.erase agent ]
Std.Ai.Tool — the tool protocol
A Tool is a name, a description, and an exec : String -> Task Error String.
The loop tells the model the available tools and asks it to reply with a small
JSON protocol ({"tool":…,"args":…} or {"answer":…}); the loop runs the
requested tool as a durable step and feeds the result back. The prompt-based
protocol is the portable floor that works on any OpenAI-compatible endpoint.
Native function-calling is also available: Provider.chatTools sends the
OpenAI tools request and returns the model's real tool_calls, and
Agent.nativeToolLoop drives it as a durable workflow (each turn and each tool
call a journalled step). A tool made with Tool.tool is advertised by name and
description with the permissive schema {"type": "object"} and receives
free-form JSON arguments.
Typed tools and structured answers
Tool.typed name description codec exec declares a tool's arguments with a
Codec. The model is sent Codec.jsonSchema codec as the tool's
parameters, and the arguments are decoded with the same codec before exec
runs. Arguments that do not decode never reach exec: they are a Decode
error, and both agent loops hand it back to the model as the tool's result, so
it can call again. The prompt-based loop lists a typed tool's schema and takes
its args as a JSON object.
forecast =
Tool.typed "forecast" "The forecast for a city"
(Codec.auto { city = "", days = 0 })
(\args -> Weather.forecast args.city args.days)
A structured answer works the same way. Provider.responseShape name codec is
the shape, Provider.chatStructured shape options provider messages sends it as
the OpenAI-compatible response_format ({"type": "json_schema", ...}, strict
when the schema is exact) and decodes the answer into the codec's type.
Agent.structuredToolLoop shape ... is nativeToolLoop with a typed final
answer: an answer that does not decode is handed back to the model with the
reason, and the run's output is the decoded value.
replyShape = Provider.responseShape "reply" (Codec.auto { answer = "", refund = False })
Provider.chatStructured replyShape Provider.defaultOptions provider messages
|> Task.map (\r -> r.value.refund)
OpenAI, vLLM and llama.cpp honour response_format, so compatible, an
Endpoint slot and a Local engine all work. A custom or customTools
provider receives only the messages or the tool specs, so a shaped call fails
there with InvalidSetting (setting responseShape) instead of returning free
text; Provider.customToolRequest receives the whole ToolRequest (the tool
schemas and the responseFormat) for a backend that can honour it. An answer
that does not match the shape is a Decode error naming the shape, never a
panic. Lower-level calls: Provider.chatToolRequest / measureToolRequest
with a ToolRequest, decodeStructured, and Codec.jsonSchema itself.
Std.Ai.Policy — the action firewall
This is where "the LLM is never the authorisation boundary" becomes structural.
The model does not call an effect; it proposes a typed Action (a stable
capability name, its arguments, a risk tag). A Policy is a pure Action -> Decision. Only Allow runs the effect; Deny and NeedsApproval short-circuit
without running it:
firewall act = Policy.requireApprovalAbove Policy.Low act
Policy.gate firewall (Policy.action "refund" orderId Policy.High) (Payments.refund orderId)
-- => Pending "…" — a High-risk action; the effect did NOT run
A NeedsApproval decision is a suspension point: pair it with
Durable.awaitSignal so a human approval resumes the workflow exactly where it
paused. The policy is ordinary Sky code — reviewable, testable, and diffable in a
pull request, unlike a system prompt.
Std.Ai.Trace — the token + cost ledger
Every model call records a row: the run it belongs to, the model, the token
counts, and the cost priced in Std.Money as an exact decimal string. totalCost
sums a run's per-call costs in Sky over Std.Decimal — never a float SUM in
SQL, because a float would corrupt the total, which is the whole reason cost lives
in Std.Money. The rows are the per-run / per-tenant cost figure and the audit
record of what the agent did. From v0.28.0 a row also records what the call did
(slot, tier, adapter, queue wait, latency, time to first token, tokens per
second, outcome), and training and eval runs record their own rows: see
../ai/trace.md.
Std.Ai.Memory.Pg — long-term memory
Long-term memory on PostgreSQL + pgvector (which ships in the embedded bundle).
remember upserts a passage with its embedding; search returns the passages
closest to a query by cosine distance (pgvector <=>); hybridSearch blends the
vector distance with a keyword score, so an exact term match is not lost to a
merely-close embedding. The caller supplies the embedding vector, so the module
owns storage and ranking, not the embedding model. It is PostgreSQL-only by design
(the .Pg suffix says so).
Testing offline
Every model and HTTP boundary is Sky.Core.Http, so the mock-by-default test mode
(SKY_TEST_MODE=1 + .env.test + tests/mocks/*.json) answers the LLM and any
integration call with no network and no credentials. Every Std.Ai module and the
Slack capstone are proven this way in rust/crates/sky/tests/*_flow.rs.
The capstone — examples/66-slack-agent
A 24/7 Slack bot that ties the whole stack together: an @mention starts a durable
workflow, a background poller drains it, and each run is three journalled steps —
the model call (exactly-once on resume), the Slack post behind the Policy
firewall, and a Trace cost record keyed by the run id. It ships on SQLite and
runs on embedded PostgreSQL in production unchanged. See its README.
Relationship to Std.Durable
Std.Ai adds no new Go kernels: it is pure Sky over Std.Durable and
Sky.Core.Http. Std.Durable is the general server primitive (it is not
AI-specific); Std.Ai.Agent is one workflow shape built on it. See
durable-execution.md.