Testing Sky projects

Status: the Rust compiler (rust/, cargo build --release -p sky) is the Sky compiler (the earlier compilers were removed in v0.29.0). Verified by the example sweep + compiler test suite (cargo test + xtask gates). See ../../CHANGELOG.md for the changelog. (This link pointed at ../compiler/versions.md; there is no docs/compiler/ directory.)

Sky ships with a first-class test framework: the Sky.Test stdlib module plus a sky test CLI command. Tests are plain Sky code and benefit from the same type checker, pattern exhaustiveness, and Error system as production code.

Writing a test module

Every test module exposes a single tests : List Test value. Tests can be individual assertions or grouped into suites.

module StringTest exposing (tests)

import Sky.Core.Prelude exposing (..)
import Sky.Core.String as String
import Sky.Test as Test exposing (Test)


tests : List Test
tests =
    [ Test.test "trim removes outer spaces" (\_ ->
        Test.equal "hi" (String.trim "  hi  "))
    , Test.test "contains finds substring" (\_ ->
        Test.isTrue (String.contains "ell" "hello"))
    , Test.test "toInt rejects junk" (\_ ->
        Test.err (String.toInt "abc"))
    ]

The (\_ -> ...) thunk wraps each assertion so a panic in one test doesn't abort the rest of the suite.

Assertions

From Sky.Test:

FunctionUse
equal : a -> a -> TestResultstrict equality on primitives / records / ADTs
notEqual : a -> a -> TestResultnegation
ok : Result e a -> TestResultasserts Ok _
err : Result e a -> TestResultasserts Err _
expectErrorKind : ErrorKind -> Result Error a -> TestResultasserts specific kind
isTrue : Bool -> TestResultasserts True
isFalse : Bool -> TestResultasserts False
fail : String -> TestResultunconditional failure with message
pass : TestResultunconditional pass

Running tests

# From your project root (containing sky.toml):
sky test tests/MyTest.sky

# Several suites in one run (the exit code is the worst of them):
sky test tests/MyTest.sky tests/OtherTest.sky

# Or from any directory:
cd tests && sky test Core/CoreTest.sky

sky test <file> builds that suite and the project modules it imports, and nothing else: a type error in another suite under tests/, or in an app module the suite does not import, does not stop it. sky verify runs every suite.

Exit code:

Output format:

  ok    String.trim
  ok    String.toUpper
  FAIL  String.split non-empty
          expected True, got False
5 passed, 1 failed (6 total)

A failing Test.equal / notEqual / ok / err prints its values in Sky syntax, with the printer toString uses (a String quoted):

  FAIL  a Result that differs
          expected Ok "a" but got Ok "b"
  FAIL  an Error that differs
          expected Err (Io "x") but got Ok 1

Machine-readable output (SKY_TEST_JSON)

Set SKY_TEST_JSON to a path and the run additionally writes a per-case JSON report. The human output above is byte-identical either way, so turning this on never changes what you read in the terminal.

SKY_TEST_JSON=/tmp/report.json sky test tests/MyTest.sky
{
  "schema": "sky-test/v1",
  "cases": [
    { "name": "String.trim", "outcome": "pass", "assertions": 1, "message": "" },
    { "name": "String.split non-empty", "outcome": "fail", "assertions": 1,
      "message": "expected True, got False" }
  ],
  "total": 6, "passed": 5, "failed": 1, "assertions": 6
}

name is the fully-qualified leaf name, so suite labels appear exactly as the human summary prints them. This is what lets a CI gate attribute a failure to a specific test case rather than to a whole suite, and what lets it assert an exact case count so a suite that silently stops running cases fails instead of passing with fewer.

Two limits, so they are not mistaken for bugs:

Sky.Test.jsonReport (pure, List (String, TestResult) -> String) and Sky.Test.writeJsonReport (a Task, no-op on an empty path) are exposed for callers that drive Sky.Test.run themselves.

Coverage — sky test --coverage

sky test --coverage records which top-level functions and which case arms of your Sky modules the suites ran. sky coverage report shows the result per module.

sky test tests/ShapeTest.sky tests/SquareTest.sky --coverage   # data in .sky-coverage/
sky test tests/ShapeTest.sky --coverage=build/cov              # data in build/cov/
sky coverage report                                            # text, per function
sky coverage report --by-module                                # text, one line per module
sky coverage report --format json                              # the JSON below
sky coverage report build/cov --all                            # every module, not only the app's

sky test takes one or more suites. With --coverage each suite is built as a coverage build and runs as usual, and the run ends with one summary line on stderr:

coverage: 1 module(s), functions 1/3 (33.3%), case arms 1/2 (50.0%)

Each sky test --coverage starts the coverage directory afresh, so a report covers the suites of one invocation. The default directory is .sky-coverage/ in the project (sky init lists it in .gitignore). The report merges the suites: a function or arm counts as run when any suite ran it.

What is counted:

By default the report shows the project's own modules (not tests/, not Sky dependencies, not the stdlib). --all adds the test suites, dependencies and the stdlib functions the program reached.

The text report prints each module, then each function that never ran or has an arm that never ran:

coverage: 1 module(s), functions 1/3 (33.3%), case arms 1/2 (50.0%)
src/Shape.sky  Shape  functions 1/3  arms 1/2
  src/Shape.sky:12 describe  ran  arms not run: #1 line 18
  src/Shape.sky:23 twice  never ran
  src/Shape.sky:28 area  never ran

--format json prints one JSON object. The shape is stable (version 1): a new field may be added, and no field is removed or renamed within a version.

{
  "version": 1,
  "runs": 1,
  "functionsHit": 1, "functionsTotal": 3, "armsHit": 1, "armsTotal": 2,
  "modules": [
    {
      "module": "Shape",
      "file": "src/Shape.sky",
      "source": "app",
      "functionsHit": 1, "functionsTotal": 3, "armsHit": 1, "armsTotal": 2,
      "functions": [
        { "name": "describe", "line": 12, "hit": true,
          "arms": [ { "index": 0, "line": 15, "hit": true },
                    { "index": 1, "line": 18, "hit": false } ] },
        { "name": "twice", "line": 23, "hit": false, "arms": [] },
        { "name": "area", "line": 28, "hit": false, "arms": [] }
      ]
    }
  ]
}

runs is the number of suites merged. source is app, test, dependency or stdlib. A function's line is the line of its name, an arm's line the line of its pattern. Modules are sorted by name, functions by line, arms by index. (The JSON printer sorts the keys of each object.)

How it works, and why there are no //line directives: a coverage build records, while it emits main.go, where each function body and each arm starts in main.go, and writes that map as sky-coverage-map.json beside main.go. The emitted main.go is byte-identical to an ordinary build. The build adds go build -cover -coverpkg=sky-app, which instruments the program and not the Go runtime, and each suite binary writes its counters to its own GOCOVERDIR (.sky-coverage/runs/<n>-<suite>/covdata/). sky coverage report turns the counters into a profile with go tool covdata textfmt and joins its blocks to the map by main.go line and column. Sky does not write //line file.sky:N directives into main.go. The cover tool cannot carry Sky line numbers through them: with a relative file name it takes the file name but keeps the Go line numbers (a block at main.go line 8 shows as Shape.sky:8), and with an absolute path it reports main.go. A //line directive would also remap go build errors and panic traces, which the build maps back to Sky by their main.go line.

A suite binary that is killed before it exits writes no counters, and its run counts as nothing ran. sky test --coverage --format json takes one suite per run.

Module discovery

sky test synthesises an entry module that imports your test module and calls Sky.Test.runMain tests. The synthesis derives the module name from the path:

The test file's module declaration must match this derived name. Directory segments are auto-capitalised (tests/core/ → Core.).

Testing Result / Error

Test.test "network errors carry retry hint" (\_ ->
    case Http.get "https://example.com/down" of
        Ok _ ->
            Test.fail "expected failure"

        Err e ->
            Test.isTrue (Error.isRetryable e)
    )

For Result Error a values, expectErrorKind is concise:

Test.test "a short password returns InvalidInput" (\_ ->
    Test.expectErrorKind InvalidInput
        (Auth.passwordStrength "short"))

Example-level verification

For end-to-end verification of example projects (build + run + panic detection + HTTP probe), sky verify is the harness. Use sky test for unit-level stdlib and app logic; use sky verify for full-stack example regression.

Test mode: offline effects and mock-by-default

A scenario test needs the app's effects to run without a live world — no real network, no shared database, a fixed clock. Sky owns its effect boundary, so it supplies that world itself. Test mode is opt-in per project: a project that has a .env.test file runs sky test in test mode; a project without one runs exactly as before, unless its .env names a database (see Ephemeral database).

In test mode the runner:

Mock-by-default outbound HTTP

With SKY_TEST_MODE on, the shared HTTP client's transport intercepts every outbound request, so no test ever touches the real network:

A fixture is one JSON file under tests/mocks/ (or the dir named by SKY_TEST_MOCKS_DIR). The shape:

{
  "match": { "method": "POST", "urlContains": "/v1/checkout/sessions" },
  "status": 200,
  "body": "{\"id\":\"cs_test_123\",\"payment_status\":\"paid\"}"
}

Fixtures load once per run and the first match wins, in filename-alphabetical order. A broad fixture (urlContains: "") shadows the ones after it, so name a specific fixture to sort before a general one (00-charge-ok.json before 99-catch-all.json).

You do not have to memorise the shape: an unmatched request prints it. The fail-closed error names the method, the URL, the mocks dir, and the exact {match:{method,urlContains},status,body} skeleton to add.

Success, pending, failure — several outcomes for one call

Two shapes are not expressible today, and are worth knowing before you design around them: two responses told apart by request query/headers alone (the matcher reads the method, the URL and the body), and a sequenced response (first call pending, second succeeded). Use distinct URLs, distinct bodies or separate directories until the matcher grows those fields.

Determinism (opt-in, decoupled from test mode)

These are decoupled from SKY_TEST_MODE on purpose: without a seed, Data.newId () stays unique across runs, so a DB test does not collide with itself. Opt into a seed only when you want reproducibility.

Ephemeral database

A project that declares a [database] but is given no DSN gets an offline database for the run, thrown away at the end. The engine decides how: a Postgres app gets a throwaway embedded cluster; a SQLite app has its path redirected to a scratch file (it is already offline). So a scenario needs no live database. A DSN given for the tests opts back out, and the test then targets that database. For a Postgres app that is DATABASE_URL or <PREFIX>_DB_PATH from .env.test / .env.test.local or the environment (the first that sets it wins), or a sky.toml [database] path / url, which is compiled into the app. The run prints one line that names the source (never the value). An explicit DSN wins as it does for sky run, because the runtime refuses an embedded cluster beside one. For a SQLite app only DATABASE_URL from .env.test or the environment opts out. An empty value (DATABASE_URL=, the usual way to clear one in CI or in .env.test) is no DSN: the run still gets its throwaway database.

sky test never takes a database from .env. A .env often names a shared or a production database, and a test and its seeds would write to it. So a DSN in .env is never used: the run prints one line that names it and points at .env.test for a test DSN. A project whose .env names a database runs sky test in test mode even without a .env.test, so its tests get the throwaway database. sky fuzz does the same. System.loadEnv () loads nothing in test mode, and a file a program names explicitly still loads.

Test mode is the whole of test mode, not only the database. A project whose .env names a database therefore runs its tests with outbound HTTP mocked: a request with no fixture fails closed with an Http error. The clock and the random seed stay real unless .env.test sets SKY_TEST_CLOCK_MS or SKY_TEST_SEED (see Determinism).

The run stays out of test mode only when the environment sets every DSN name .env sets (<PREFIX>_DB_PATH and DATABASE_URL) and the project has no .env.test. One name the environment leaves unset is enough to turn test mode on, because the runtime would load that name from .env. So to make live HTTP calls from a test, set each of those names in the environment for the run (to a database you are willing to write to) and keep no .env.test. Test mode itself has no switch that sends a request to the real network.

Training runs (Std.Ai.Train)

In test mode a Std.Ai.Train run starts no trainer, installs no toolchain and downloads nothing. The durable workflow runs for real on the test database, and its launch step writes what the scripted Fake trainer writes: the protocol lines, verified checkpoints and a real, no-op PEFT adapter. The GGUF conversion and the registry steps run too. So Train.drive returns an adapter in milliseconds, and a test can assert the steps, the losses, the relaunches and the registered version.

Put SKY_AI_TRAIN_BACKEND=fake in .env.test, so the test does not depend on the host's accelerator. Train.fakeCrashAt n makes the first attempt stop after step n with no terminal line, so a test sees the relaunch from the newest checkpoint. examples/73-ai-train/tests/TrainExampleTest.sky trains, crashes, resumes, registers and serves an adapter this way. See ../ai/train.md.

Google Cloud (Std.Ai.Cloud.Google, Train.Vertex, Std.Ai.Serve)

Put SKY_AI_GCP_ENDPOINT=http://sky-gcp.invalid in .env.test. In test mode every Google Cloud call to that host is answered in process by an emulator in the runtime: OAuth and the metadata server, Cloud Storage (with generation preconditions), Vertex AI custom jobs (which run the Fake trainer's protocol into the staging bucket), Cloud Run services and their operations, IAM, Artifact Registry, Cloud Build, Cloud Quotas, the Cloud Billing catalogue and Cloud Logging. Its state lives for the test process, so a test can launch, watch, cancel, deploy and tear down, and assert what is left. examples/74-ai-cloud/tests/CloudTest.sky trains on Vertex AI, registers, deploys the tier and calls it this way.

When several processes must share one emulated project (an app and the sky ai verbs, as the cargo flows do), run runtime-go/rt/testdata/fakegcp on a loopback port and point SKY_AI_GCP_ENDPOINT at it. Test mode lets requests to that one loopback host through and keeps every other request mocked.

The paid tests on a real project are rust/crates/sky/tests/ai_gcp_live.rs. They need Need::GcpLive: they fail unless the wrapper set SKY_AI_GCP_LIVE=1 (or SKY_LIVE_TESTS=skip). Run them only through scripts/ai/gcp-live.sh, which checks the cost ledger before it starts, labels every resource, tears everything down on exit and fails the run if anything is left. CI runs them on manual dispatch only (.github/workflows/ai-cloud-live.yml), never on a commit or a release.

On-device slots (Std.Ai.Device)

A test runs on the server, so Device.provider "<slot>" is the slot itself: its calls are answered by tests/mocks/ like any slot's, Device.status reports ServerSide OnTheServer, and Device.recordPrompts with Eval.prompts followed by Device.replay and Eval.runAs scores a recording offline. examples/75-ai-everywhere/tests/NotesTest.sky does all three. The device itself (the browser worker, the iOS and Android apps, the Mac app) is tested by scripts/ai-device-e2e.sh and rust/crates/sky/tests/ai_device_native_flow.rs on a synthetic model (go run ./cmd/sky-tiny-gguf <dir> in runtime-go/), set through the slot's own variables (SKY_AI_<SLOT>_MODEL). See ../ai/device.md.

Log capture

Set SKY_TEST_LOG_CAPTURE=<path> and the app's Std.Log output is written to that file. A scenario reads it back with File.readFile to assert on what the app logged — or to recover a value the app would otherwise only have emailed, for example a verification code, which makes the "check your email" step readable in-process.

A verification-flow scenario, offline

These features compose into a real account-creation + code-verification test with no external world. Signup runs under a fixed SKY_TEST_SEED, so the generated code is reproducible; the outbound "send email" call is intercepted by a fixture (or read back from the ephemeral DB / the log capture); the test then submits the code and asserts the account is active. The only thing that stays mocked is a real message actually delivered by a third party — automated tests prove the app's handling of the flow, never a provider's delivery.

AI models in test mode, and sky ai --test-mode

The step-by-step guide for the whole AI lifecycle, from the offline checks to the paid live runs, is ../ai/testing-guide.md.

A model slot (Std.Ai.Endpoint) is mocked like any other outbound call: in test mode a local slot's calls go to http://sky-ai-local.invalid/<slot>/v1/… and are answered by tests/mocks/ fixtures (match.bodyContains picks the answer per question, model or topic), so no engine starts and nothing is downloaded. An eval suite is then an ordinary test: Pipeline.evalSuite Ai.pipeline "support" |> Task.run |> Test.ok (../ai/pipeline.md).

The project-mode sky ai verbs take --test-mode: they load .env.test (and the project's .env under it) and answer model calls from the same fixtures, so CI can run sky ai eval <suite> --test-mode and sky ai data freeze <dataset> --test-mode with no engine. Unlike sky test, a sky ai verb keeps the project's own database (it acts on the project's data), unless --db names another. examples/72-ai-support-desk runs its whole lifecycle this way.

Property-based fuzzing — sky fuzz [--target <t>]

One command derives its inputs from the app's own types, so it needs no hand-written oracle. It runs one or two nets, chosen by --target.

sky fuzz src/Main.sky --iters 500 --seed 42          # model no-panic net
sky fuzz src/Main.sky --target web:app --iters 500   # + the differential split oracle

The model no-panic net (always, any TEA app)

sky fuzz derives a Msg generator from the app's own Msg union, folds random Msg sequences from init () through the real update, and asserts no unclassified panic. Every sequence is a valid input by construction — a client can send any Msg, in any order — so it drives exactly the hostile, out-of-order input a real client can (an EnterCode before any signup, a SubmitCode while anonymous). It works on any TEA app, Spa or Live, because it needs only (Model, Msg, update), and it runs under test mode with the offline database above, so a DB-backed app fuzzes offline. It exits 0 on PASS and non-zero on the first sequence that panics, reproducible with the same --seed.

The net is coverage-guided by default. The harness is built with an in-process counter for each case arm and each top-level function (the markers of Coverage, each compiled as rt.CovHit(<id>)). Every sequence starts from init (). After each one, the loop reads which arms it reached for the first time. For each step that reached a new arm it keeps the Msg prefix that ends at that step. Seven of eight later sequences change a kept prefix: they append one or two fresh Msgs, replace one Msg, or join the start of one kept prefix to the end of another. A prefix whose new arms are rarely reached is picked most often, so a state that needs a particular order of Msgs (Open, then Turn, then Fire) is reached step by step instead of by chance. One seed drives every choice, so a run is reproducible. --iters N is the budget of Msgs folded through update, as before.

sky fuzz src/Main.sky --iters 1000 --seed 1              # guided (the default)
sky fuzz src/Main.sky --iters 1000 --seed 1 --unguided   # fresh random sequences only
sky fuzz src/Main.sky --coverage                         # + a run `sky coverage report` reads

A guided run prints case arms reached R/T: the arms of your modules (not the harness's own code) that the run reached. The harness writes the ids it reached to the file SKY_FUZZ_COV_OUT names, which sky fuzz sets for it (an internal handoff, not a setting). When a sequence panics, the net shrinks it to the fewest Msgs from init () that still panic and prints them before the panic report:

sky fuzz: panicking sequence (3 Msg(s) from init): Open, Turn, Fire

--coverage (data in .sky-coverage/) or --coverage=<dir> also builds the harness with go build -cover and records one run, so sky coverage report shows which functions and arms the fuzz reached, in the same JSON shape as sky test --coverage. The harness's generated definitions are left out.

The differential split oracle (Sky.Spa client targets)

When --target selects a client/server split (a Sky.Spa wasm client: web:app, mobile*, desktop:<os>, tablet:<os>; the default is the project's sky.toml [app] target), sky fuzz ALSO runs each random (Model, Msg) two ways — directly, and through the client/server split plumbing (build the request from the read-set, reconstruct the server model, apply the write-set delta) — and asserts the two results agree. A dropped read or a Msg-argument collision diverges the two paths and is caught mechanically, with no oracle and no real credentials. A non-split target (or none) runs the model net alone; a whole update app with no resolvable case msg of skips the oracle with a note rather than failing. (This replaces the former sky spa-diff-fuzz verb.)

Scaffolding the fixtures — sky test --scaffold-mocks

You do not have to hand-write the fixtures. The compiler knows which outbound calls the app makes, and to which URLs, because that is in the typed IR. Run:

sky test --scaffold-mocks

It walks the project's own modules for outbound Sky.Core.Http calls and writes one skeleton per distinct call under tests/mocks/, with match.method and match.urlContains pre-filled. It recovers the URL from the shapes real code uses, not only a bare literal: a direct Http.get "https://…", a piped … |> Http.request builder, and a base ++ "/path" concat (whose literal path suffix becomes a host-independent urlContains). It never overwrites an existing fixture, and a call whose URL is fully computed gets a blank urlContains for you to narrow.

You fill one thing: the response body. Paste a captured real payload there. A captured payload is the most faithful body, because it proves the app's decoder handles the shape the provider actually sends, not only the shape the app models.

The fixture schema above plus the self-documenting fail-closed error are enough to hand-write or adjust a mock by hand when you prefer.

Regression discipline

Every bug that reaches production gets a permanent regression test:

  1. Reproduce with a minimal Sky fixture.
  2. Add the failing test under tests/ (or as a Rust cargo test spec / runtime-go/rt/*_test.go if the bug lives in the compiler or runtime).
  3. Fix the root cause.
  4. Verify the regression test passes with the fix and fails without it.

Current permanent regressions:

Known limits