Testing Sky projects

Status: the Rust compiler (rust/, cargo build --release -p sky) is the primary Sky compiler; the Haskell compiler is preserved under legacy-haskell-compiler/. Verified by the example sweep + compiler test suite (cargo test + xtask gates). See ../../CHANGELOG.md for the changelog. (This link pointed at ../compiler/versions.md; there is no docs/compiler/ directory.)

Sky ships with a first-class test framework: the Sky.Test stdlib module plus a sky test CLI command. Tests are plain Sky code and benefit from the same type checker, pattern exhaustiveness, and Error system as production code.

Writing a test module

Every test module exposes a single tests : List Test value. Tests can be individual assertions or grouped into suites.

module StringTest exposing (tests)

import Sky.Core.Prelude exposing (..)
import Sky.Core.String as String
import Sky.Test as Test exposing (Test)


tests : List Test
tests =
    [ Test.test "trim removes outer spaces" (\_ ->
        Test.equal "hi" (String.trim "  hi  "))
    , Test.test "contains finds substring" (\_ ->
        Test.isTrue (String.contains "ell" "hello"))
    , Test.test "toInt rejects junk" (\_ ->
        Test.err (String.toInt "abc"))
    ]

The (\_ -> ...) thunk wraps each assertion so a panic in one test doesn't abort the rest of the suite.

Assertions

From Sky.Test:

FunctionUse
equal : a -> a -> TestResultstrict equality on primitives / records / ADTs
notEqual : a -> a -> TestResultnegation
ok : Result e a -> TestResultasserts Ok _
err : Result e a -> TestResultasserts Err _
expectErrorKind : ErrorKind -> Result Error a -> TestResultasserts specific kind
isTrue : Bool -> TestResultasserts True
isFalse : Bool -> TestResultasserts False
fail : String -> TestResultunconditional failure with message
pass : TestResultunconditional pass

Running tests

# From your project root (containing sky.toml):
sky test tests/MyTest.sky

# Or from any directory:
cd tests && sky test Core/CoreTest.sky

Exit code:

Output format:

  ok    String.trim
  ok    String.toUpper
  FAIL  String.split non-empty
          expected True, got False
5 passed, 1 failed (6 total)

Machine-readable output (SKY_TEST_JSON)

Set SKY_TEST_JSON to a path and the run additionally writes a per-case JSON report. The human output above is byte-identical either way, so turning this on never changes what you read in the terminal.

SKY_TEST_JSON=/tmp/report.json sky test tests/MyTest.sky

{
  "schema": "sky-test/v1",
  "cases": [
    { "name": "String.trim", "outcome": "pass", "assertions": 1, "message": "" },
    { "name": "String.split non-empty", "outcome": "fail", "assertions": 1,
      "message": "expected True, got False" }
  ],
  "total": 6, "passed": 5, "failed": 1, "assertions": 6
}

name is the fully-qualified leaf name, so suite labels appear exactly as the human summary prints them. This is what lets a CI gate attribute a failure to a specific test case rather than to a whole suite, and what lets it assert an exact case count so a suite that silently stops running cases fails instead of passing with fewer.

Two limits, so they are not mistaken for bugs:

Sky.Test.jsonReport (pure, List (String, TestResult) -> String) and Sky.Test.writeJsonReport (a Task, no-op on an empty path) are exposed for callers that drive Sky.Test.run themselves.

Module discovery

sky test synthesises an entry module that imports your test module and calls Sky.Test.runMain tests. The synthesis derives the module name from the path:

The test file's module declaration must match this derived name. Directory segments are auto-capitalised (tests/core/ → Core.).

Testing Result / Error

Test.test "network errors carry retry hint" (\_ ->
    case Http.get "https://example.com/down" of
        Ok _ ->
            Test.fail "expected failure"

        Err e ->
            Test.isTrue (Error.isRetryable e)
    )

For Result Error a values, expectErrorKind is concise:

Test.test "unauthorised returns PermissionDenied" (\_ ->
    Test.expectErrorKind PermissionDenied
        (Auth.authenticateUser "bad@email" "wrong-password"))

Example-level verification

For end-to-end verification of example projects (build + run + panic detection + HTTP probe), sky verify is the harness. Use sky test for unit-level stdlib and app logic; use sky verify for full-stack example regression.

Regression discipline

Every bug that reaches production gets a permanent regression test:

  1. Reproduce with a minimal Sky fixture.
  2. Add the failing test under tests/ (or as a Rust cargo test spec / runtime-go/rt/*_test.go if the bug lives in the compiler or runtime).
  3. Fix the root cause.
  4. Verify the regression test passes with the fix and fails without it.

Current permanent regressions:

Known limits