Sky.Live input authority protocol

Status: the Rust compiler (rust/, cargo build --release -p sky) is the primary Sky compiler; the Haskell compiler is preserved under legacy-haskell-compiler/. Verified by the example sweep + compiler test suite (cargo test + xtask gates). See ../compiler/versions.md for the changelog.

Design spec for the reliability layer that guarantees no keystroke is ever lost, regardless of latency, race conditions, dropped requests, out-of-order responses, page navigation, or server restarts. Supersedes the purely-positional morph algorithm documented in architecture.md.

Status: partially implemented (in progress). Recent commits on main have landed __skyMorph retirement (atomic innerHTML + focus snapshot/restore), DOM-node preservation through innerHTML swap, dirty-input scoping to typable form fields only, and dispatch-error narrowing. The full sky-id grammar / monotonic seq protocol below is still partially open work — check git log -- runtime-go/rt/live.go for current status.

Motivation

The current runtime has three structural weaknesses:

  1. Positional sky-ids collide across pages. assignSkyIDs in live.go:836 emits r.0.1.2 — purely the walk index. Two pages that share an outer wrapper (<div class="container">) but diverge internally end up with matching ids at structurally different elements. The diff at diffNodes in live.go:911 recurses into them, tries to patch, and emits wrong HTML.

  2. No patch preservation for focused inputs on the JSON path. __skyApplyPatches in live.go:2160 does el.innerHTML = p.html blindly. If the server sends an innerHTML patch at any ancestor of a focused input, the input is wiped mid-keystroke. Only the __skyMorph path (used for full-HTML responses, not the common event path) has any focus protection, and it only covers same-tag matches.

  3. No ordering or replay protection. Two fetches in flight can return in reversed order; the client applies both in arrival order. The older response clobbers the newer.

Under mild load on a downstream app, this produces visible input duplication on page navigation and occasional keystroke loss. Under adversarial conditions (slow 3G, tab-switching while typing, concurrent submits) it becomes unusable.

The fix is a coordinated protocol change: stable structural identities, monotonic sequence numbers, and client-side authority for dirty DOM state.

Guarantees

The protocol guarantees, for any Sky.Live app recompiled against the new runtime:

Explicitly not guaranteed (out of scope for v1):

Wire format

Sky-id grammar

sky-id    = "r"                                        -- root
          | parent-id "." index "#" tag [ ":" key ]

parent-id = sky-id
index     = 1*DIGIT
tag       = 1*ALPHA                                    -- lowercase HTML tag
key       = 1*( ALPHA / DIGIT / "-" / "_" )            -- disambiguator

Key priority (first match wins):

  1. Explicit Html.keyed "k" node wrapper → k.
  2. name attribute on input, textarea, select, form, button, fieldset → the attribute value.
  3. No key.

Examples:

VNodesky-id
root <div id="sky-root">r
first child <div> (CSS stylesheet)r.0#div
<header> at index 1r.1#header
Email input inside a formr.2#main.0#form.3#input:email
Keyed list itemr.4#ul.0#li:todo-42

Properties:

Request: POST /_sky/event

The session cookie is the authority. The server resolves the session only from the session cookie — sky_sid for the host app (sub-apps mounted in-process use sky_<name>_sid). The sessionId in the body is advisory: it may agree with the cookie or be omitted, but it can never select a different session. A request whose body names a session other than the cookie's, or that carries no session cookie at all, is rejected with the same 404 + X-Sky-Status: session-lost as an unknown session — the endpoint deliberately gives no signal about which session ids exist.

The browser satisfies this automatically: the in-page fetch uses credentials: "same-origin", and navigator.sendBeacon (the unload batch flush) rides the same cookie jar. GET /_sky/sse has always required the cookie, so any client with a live connection already sends it. Non-browser clients must send the cookie explicitly.

CSRF token on the beacon. sendBeacon cannot set headers, so the unload batch carries the CSRF token as a csrf field in its JSON body instead of the X-Sky-Csrf header. The server compares it to the __sky_csrf cookie exactly as it does the header. Every other client path uses the header.

{
  "sessionId": "3f2a8b...",
  "seq": 42,
  "msg": "UpdateEmail",
  "args": ["alice@example.com"],
  "handlerId": "r.2#main.0#form.3#input:email.input",
  "inputState": {
    "r.2#main.0#form.3#input:email": {
      "value": "alice@example.com",
      "seq": 42
    },
    "r.2#main.0#form.5#input:password": {
      "value": "hunter2",
      "seq": 40
    }
  }
}

New fields:

Backward compat: old clients omit seq and inputState. Server treats missing seq as 0 (oldest), missing inputState as empty.

Response: event reply

When the server has usable patches to send back as JSON:

{
  "seq": 42,
  "ackInputs": {
    "r.2#main.0#form.3#input:email": 42,
    "r.2#main.0#form.5#input:password": 40
  },
  "patches": [
    {"id": "r.2#main.0#form.3#input:email", "attrs": {"value": "alice@example.com"}},
    {"id": "r.1#header.0#span:greeting", "text": "Hi, Alice"}
  ]
}

New fields:

When the server needs to send a full HTML body (first interaction, or patches all target root):

Batched events: POST /_sky/event with batch field

Used by __skyFlushAllPendingSync on tab unload (via navigator.sendBeacon). Same endpoint as single events — no second route:

{
  "sessionId": "3f2a8b...",
  "batch": [
    {"seq": 42, "msg": "UpdateEmail", "args": ["a@b.com"], "handlerId": "..."},
    {"seq": 43, "msg": "UpdatePhone", "args": ["+44..."],  "handlerId": "..."}
  ]
}

Server processes batch entries sequentially under sess.mu, dispatching each Msg as if it arrived on its own. Response body is ignored by sendBeacon, but the server still stamps out frames (via SSE) so other clients / tabs observe the state changes. When batch is set, top-level msg/args/handlerId/seq are ignored.

SSE frames: GET /_sky/subscribe

event: patch
data: {"seq": 99, "ackInputs": {}, "patches": [...]}

Same shape as the JSON event reply. SSE frames and event replies share the single session-wide sess.outSeq counter — every mutation of session state produces the next seq, regardless of what triggered it. The server already has a single linearisation point (sess.mu), so there's one true ordering; exposing it as one counter lets the client resolve cross-channel races correctly. A Cmd goroutine that completes during an in-flight event reply will have its SSE frame stamped after the reply if the reply's dispatch ran first, or before if the goroutine's dispatch ran first — and the client applies in that exact order.

Client state

// Top-level session state.
var __skyClientSeq = 0;             // monotonic; client-owned, tagged on every outgoing event
var __skyLastAppliedSeq = 0;        // monotonic; server-owned, drop any frame with seq ≤ this
var __skyInputs = {};               // per sky-id → InputEntry

Two counters, different directions: __skyClientSeq is what the client stamps on outgoing events (so the server can match an inputState snapshot to its producing event). __skyLastAppliedSeq tracks the server's session-wide outSeq — a single counter covering both event replies and SSE frames, because the server has a single mutation order.

interface InputEntry {
  liveValue: string;       // most recent DOM value the user typed
  lastSentSeq: number;     // seq when this value was bundled into inputState
  lastAckedSeq: number;    // largest seq the server has acked for this id
  pendingDebounceId: number | null;  // window.setTimeout handle
  pendingSend: {msgName: string; args: any[]; hid: string} | null;
}

State transitions (per input I)

EventUpdate
input fires on II.liveValue := el.value
Debounce schedules a sendI.pendingSend := {...}, I.pendingDebounceId := setTimeout(…)
__skySend flushes II.lastSentSeq := ++__skySeq, include I in req.inputState, clear pending
Response arrives with ackInputs[I.id] = NI.lastAckedSeq := max(I.lastAckedSeq, N)
Patch arrives with attrs.value on ISee "Authority rule" below
I becomes unmounted (not in new DOM)delete __skyInputs[I.id]

Authority rule (client-side patch filter)

When __skyApplyPatches processes a patch p targeting an element that is an input/textarea/select with attrs.value, attrs.checked, or attrs.selected:

entry = __skyInputs[p.id]
el    = querySelector([sky-id=p.id])

if entry exists AND (el is focused OR entry.pendingDebounceId != null):
    DROP the value/checked/selected keys from p.attrs
    // apply the rest of the patch (class, style, aria-*, etc.)

else if entry exists AND p.attrs.value == entry.liveValue:
    DROP (server is echoing our own value — no-op)

else if response.seq <= entry.lastAckedSeq:
    DROP the whole patch (stale)

else:
    APPLY
    entry.liveValue = p.attrs.value   // realign

The same filter runs for innerHTML patches targeting ancestors of a focused/dirty input: an innerHTML patch that would wipe a dirty input is rewritten to a scoped patch that preserves the input's subtree. Implementation: before applying innerHTML, scan the new HTML for sky-ids, diff against existing dirty inputs, and if a dirty input is inside the target, fall back to per-element patching for that subtree (morph-style).

This preserves G1 even when the server sends a "wipe your whole form and rebuild" patch.

Server state

type liveSession struct {
    // ... existing fields ...

    // Single monotonic counter for every outgoing frame (event reply OR
    // SSE patch). Bumped under sess.mu, so reflects the session's actual
    // linearisation order. Client uses this as the authoritative sequence.
    outSeq int64

    // Per-input-id → largest client seq the server has observed.
    // Used to populate ackInputs on every response. Evicted when the
    // corresponding input is no longer in the rendered tree.
    inputSeqs map[string]int64
}

Why one counter not two: every state mutation (dispatching a user event, completing a Cmd goroutine, applying a subscription tick) passes through dispatch() which holds sess.mu. There's exactly one true order. A single counter surfaces that order to the client; two counters would let the client accidentally reorder unrelated mutations that the server had already serialised.

Server-side input reconciliation — fused into diff

Rather than pre-walk prev to rewrite values (O(n) extra pass, requires an index to be efficient at scale), the client's inputState is passed directly into diffNodes as a lookup table. The diff already walks the tree — we just consult the lookup at input nodes:

func diffNodes(old, new_ *VNode, clientState map[string]string, out *[]Patch) {
    // ... existing tag/kind check, attr diff ...

    // When diffing an input-tag value/checked/selected attr, treat the
    // client-reported value as the effective old value. This prevents the
    // diff from emitting a patch that reverts the user's in-progress typing.
    if isInputTag(new_.Tag) {
        if cv, ok := clientState[old.SkyID]; ok {
            if new_.Attrs["value"] == cv {
                delete(attrChanges, "value")  // server matches DOM — no patch
            }
            // else: server genuinely wants to overwrite (form reset, etc.)
            //       emit the patch; Step 3 client filter decides whether to apply
        }
    }

    // ... recurse into children with same clientState ...
}

On every POST /_sky/event:

1. Read req.seq, req.inputState.
2. For each (id, {value, seq}) in req.inputState:
     if sess.inputSeqs[id] < seq:
         sess.inputSeqs[id] = seq
3. Build clientState map[sky-id]string from req.inputState.
4. Dispatch Msg as usual.
5. Call diffNodes(prev, new_, clientState, &patches).
6. Reply with {seq: ++sess.outSeq, respondingTo: req.seq,
               ackInputs: subsetOfInputSeqsInPrevTree, patches}.

Why fused, not pre-pass: the diff already walks both trees. Adding a pre-pass walks the tree twice for no benefit. Adding an index on prev means maintaining invalidation on every mutation — another rule to get wrong. The fused approach costs O(hash-lookup) per input node, zero extra allocations, and no state to invalidate.

What "client authority" means on the server: the diff is a hint, not a veto. If update decides model.email = "Bob" (form reset, admin edit, whatever), new_.Attrs["value"] becomes "Bob", which doesn't match the client's reported value, so the diff emits value="Bob". The Step 3 client filter then decides — if the user is currently typing into that field, the filter drops the value attr and the user keeps typing; otherwise the patch applies. Server model stays authoritative; client input stays unclobbered. Both invariants hold simultaneously.

Invariants enforced

InvariantMechanismLocation
I1 — Client authorityPatch filter drops value/checked/selected when input is focused/dirty__skyApplyPatches, __skyMorph
I2 — Monotonic applicationresponse.seq ≤ lastApplied → drop__skyApplyPatches, SSE handler
I3 — Navigation flushIntercept <a> click inside sky-root → flush debounces synchronously → send nav Msg__skyBindEvents, beforeunload
I4 — Identity collision-freeStructural sky-ids with tag+keyassignSkyIDs in live.go
I5 — Server diff uses real DOMalignPrevTreeValue merges req.inputState into prev before diffing/_sky/event handler

Failure-mode matrix

ScenarioBeforeAfterInvariant
Type fast, server responds slow with old valueInput reverts to old value mid-typePatch dropped; user's value keptI1 + I5
Two sends in flight, responses arrive reversedOlder response clobbers newerOlder patch dropped via seq checkI2
Click <a> while typingFinal keystrokes lostDebounce flushed before nav leavesI3
Navigate to page with same outer wrapperInputs duplicate / wrong attrsDifferent sky-ids; clean replaceI4
SSE delivers duplicate frameFrame applied twiceSecond frame dropped via seqI2
Form reset via update sets model.email = ""Works today, still works aftervalue="" patch applied; empty input takes effect(no regression)
User types, tab-switches, comes backWorks if debounce already flushed via blurSame: focusout handler flushesI3 (existing)
Server sends innerHTML at parent of focused inputInput wiped, value lostScoped morph preserves dirty input subtreeI1
Slow-3G + rapid typingCharacters skip or doubleSmooth; buffered locally, flushed in seq orderI1 + I2

Implementation plan

Ordered; each step is independently landable and reviewable.

Step 1 — Structural sky-ids (Fix B)

File: runtime-go/rt/live.go

Replace assignSkyIDs (line 836) with:

func assignSkyIDs(n *VNode, path string) {
    if n.Kind != "element" {
        return
    }
    seg := path
    if path == "r" {
        n.SkyID = "r"
    } else {
        n.SkyID = path
    }
    for i := range n.Children {
        child := &n.Children[i]
        childSeg := seg + "." + itoa(i) + "#" + child.Tag
        if k := skyIDKey(child); k != "" {
            childSeg += ":" + k
        }
        assignSkyIDs(child, childSeg)
    }
}

func skyIDKey(n *VNode) string {
    if k, ok := n.Attrs["sky-key"]; ok && k != "" {
        return sanitiseKey(k)
    }
    switch n.Tag {
    case "input", "textarea", "select", "form", "button", "fieldset":
        if k, ok := n.Attrs["name"]; ok && k != "" {
            return sanitiseKey(k)
        }
    }
    return ""
}

func sanitiseKey(s string) string {
    // Replace anything that's not [A-Za-z0-9_-] with _.
    // Prevents sky-id parse ambiguity and HTML injection.
    var b strings.Builder
    for _, r := range s {
        switch {
        case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '-', r == '_':
            b.WriteRune(r)
        default:
            b.WriteByte('_')
        }
    }
    return b.String()
}

Also: the keyed helper. It shipped as Std.Ui.Keyed (sky-stdlib/Std/Ui/Keyed.sky:39-45), not as Html.keyed in sky-stdlib/Sky/Html.sky — that file does not exist (sky-stdlib/Sky/ holds Core/, Http/, Test.sky). Keyed.keyed injects the sky-key attribute through an outer Ui.el via Ui.htmlAttribute, consumed by skyIDKey above. The sketch below is the shape, not the shipped signature:

keyed : String -> VNode -> VNode
keyed k node = Html.attr "sky-key" k node

Verification: scripts/example-sweep.sh must pass. Every example renders; HTML diff against main shows only sky-id format changes (longer ids, no layout change).

Step 2 — Wire format bump

Files: runtime-go/rt/live.go (Go), embedded JS at line 2030+.

Add seq, inputState, ackInputs fields to request/response structs. Populate but don't yet enforce — behaviour identical to today. This is the "stub" step that makes the protocol upgradable.

Go struct additions:

type eventRequest struct {
    SessionID  string                    `json:"sessionId"`
    Seq        int64                     `json:"seq,omitempty"`
    Msg        string                    `json:"msg"`
    Args       []any                     `json:"args,omitempty"`
    HandlerID  string                    `json:"handlerId,omitempty"`
    InputState map[string]inputStateEntry `json:"inputState,omitempty"`
}
type inputStateEntry struct {
    Value string `json:"value"`
    Seq   int64  `json:"seq"`
}
type eventResponse struct {
    Seq        int64            `json:"seq,omitempty"`
    AckInputs  map[string]int64 `json:"ackInputs,omitempty"`
    Patches    []Patch          `json:"patches"`
}

Client JS:

Step 3 — I1 client authority

Files: embedded JS at line 2030+ in live.go.

Wire the patch filter. __skyApplyPatches consults __skyInputs for every patch targeting an input/textarea/select:

function __skyApplyPatches(resp) {
  var patches = resp.patches || [];
  // Ingest ackInputs first so the filter can see the latest acks.
  if (resp.ackInputs) {
    var ids = Object.keys(resp.ackInputs);
    for (var k = 0; k < ids.length; k++) {
      var e = __skyInputs[ids[k]];
      if (e) e.lastAckedSeq = Math.max(e.lastAckedSeq, resp.ackInputs[ids[k]]);
    }
  }
  for (var i = 0; i < patches.length; i++) {
    var p = patches[i];
    var el = document.querySelector('[sky-id="' + p.id.replace(/"/g, '\\"') + '"]');
    if (!el) continue;
    __skyFilterPatch(p, el);    // mutates p in place per authority rule
    __skyApplyOne(p, el);
  }
  __skyBindEvents(document);
}

__skyFilterPatch implements the authority rule from the client state section. __skyApplyOne holds the existing innerHTML / textContent / attr application logic.

For innerHTML patches that contain dirty inputs: parse the new HTML with a DOMParser, check whether any dirty input id is present, and if so, convert the innerHTML patch into a morph (walk new vs. old, skip dirty inputs, apply rest).

Step 4 — I2 stale drop

File: same JS block.

Activate the seq check:

function __skyApplyPatches(resp) {
  if (resp.seq !== undefined && resp.seq <= __skyLastAppliedEventSeq) {
    return;  // stale — a newer response already landed
  }
  __skyLastAppliedEventSeq = Math.max(__skyLastAppliedEventSeq, resp.seq || 0);
  // ... rest as in Step 3
}

SSE handler uses __skyLastAppliedSubSeq (separate counter) for subscription frames.

Step 5 — I3 flush on unmount

File: same JS block.

Intercept <a href> clicks inside sky-root:

document.addEventListener('click', function(ev) {
  var a = ev.target.closest && ev.target.closest('a[href]');
  if (!a) return;
  if (!document.getElementById('sky-root').contains(a)) return;
  // External links: let the browser handle, but flush first.
  if (/^https?:/.test(a.href) && a.host !== location.host) {
    __skyFlushAllPending();
    return;
  }
  // Internal nav: let Sky.Live's client-side router handle it, but flush first.
  __skyFlushAllPending();
  // Don't preventDefault — the existing router takes over.
}, true);

window.addEventListener('beforeunload', function() {
  __skyFlushAllPendingSync();  // uses navigator.sendBeacon
});

function __skyFlushAllPending() {
  var ids = Object.keys(__skyInputs);
  for (var i = 0; i < ids.length; i++) {
    var e = __skyInputs[ids[i]];
    if (e.pendingDebounceId !== null) {
      clearTimeout(e.pendingDebounceId);
      e.pendingDebounceId = null;
      if (e.pendingSend) {
        __skySend(e.pendingSend.msgName, e.pendingSend.args, e.pendingSend.hid,
                  {noLoader: true});
        e.pendingSend = null;
      }
    }
  }
}

function __skyFlushAllPendingSync() {
  var ids = Object.keys(__skyInputs);
  var batch = [];
  for (var i = 0; i < ids.length; i++) {
    var e = __skyInputs[ids[i]];
    if (e.pendingSend) batch.push(e.pendingSend);
  }
  if (batch.length === 0) return;
  navigator.sendBeacon('/_sky/event-batch', JSON.stringify({
    sessionId: __skySid, batch: batch
  }));
}

/_sky/event-batch is a new server endpoint that processes a batch of events in order before the tab closes. Best-effort — browsers guarantee sendBeacon delivery but not ordering across multiple beacons; a single batched beacon is reliable.

Step 6 — Adversarial test matrix

Go tests landed alongside Steps 1–5 (exhaustive per-step coverage was written as each step landed rather than deferred to a single catch-up step):

Step 1 — runtime-go/rt/live_skyid_test.go:

Step 2 — runtime-go/rt/live_protocol_test.go:

Step 3 — runtime-go/rt/live_authority_test.go:

Step 6 — runtime-go/rt/live_adversarial_test.go:

Browser tests deferred to a follow-up branch — adding Playwright (or chromedp) to this repo is a non-trivial tooling commitment and keeps this PR focused on the runtime contract. The JS logic (patch filter, stale-drop, beacon flush) is small and mechanically derived from the Go-side invariants that are tested; Step 7 validates end-to-end against a real downstream app for the regression the protocol was designed to fix.

Manual smoke checklist (for the follow-up automation):

  1. Type "hello" with 500ms server latency — all 5 characters appear.
  2. Type + click <a href> mid-keystroke — server logs show final value.
  3. Type on page A, navigate to page B — no orphan value, no state bleed.
  4. Fire two submits in rapid succession — server serializes, client shows consistent final state.
  5. SSE send duplicate frame — DOM not double-mutated.
  6. innerHTML patch at <form> while <input> focused — input survives with user's value.
  7. Kill server mid-fetch — client retries, no silent data loss.
  8. Throttle to slow-3G, type continuously — no character loss.

Step 7 — Verify on a downstream app

Rebuild a downstream app's compiler binary (cargo build --release -p sky in sky repo), rebuild the downstream app (sky build src/Main.sky). Manual check:

Backward compatibility

Open questions — resolved

  1. Batch endpoint: single endpoint. POST /_sky/event accepts an optional batch field; when present, top-level msg/args/handlerId/seq are ignored. One route, one mental model.

  2. Single session-wide seq counter. The server's sess.mu is the single linearisation point for all mutations; exposing it as one monotonic counter lets the client resolve cross-channel races correctly. Two counters would leak implementation detail (separate request/subscription tracking) and make cross-channel ordering ambiguous. Client tracks one __skyLastAppliedSeq.

  3. Fused client-value alignment. clientState map[string]string parameter passed to diffNodes. No pre-pass, no index to maintain, no invalidation rule. Consulted O(1) per input node during the walk the diff already performs.

  4. No patch reordering. Current DFS emission order is correct. Ancestor innerHTML patches are self-healing — they carry the full rendered subtree, so clobbered descendant patches don't corrupt DOM state (just waste client work). Step 3's filter with innerHTML-preserving-dirty-input morph handles the edge cases categorically. Emission-ordering optimisations (suppressing descendant patches when an ancestor innerHTML will be emitted) are a future perf tweak, not a correctness fix.

Approval gate

This doc is the contract. Before any of Steps 1–7 lands, the following must be confirmed:

Once approved, implementation proceeds in the step order above, one commit per step. Each commit ships its own regression test. No step is marked complete until scripts/example-sweep.sh passes clean-slate.