The shipped GC default: does the bound actually hold?
Yes, and the falsifier the parent run named is now closed. At 500 concurrent sessions on the PostgreSQL store, an app running the limit this rule derives for an e2-small peaks at 973 MB against a 1,044 MB bound, which puts the whole deployment — app, its embedded PostgreSQL, and the OS — at 59% of an e2-small, with 809 MB of headroom.
docs/perf/runs/gogc-postgres-20260816/ chose the setting. This run verifies
the setting as derived and shipped, and answers the question that one left
open: what happens when the live heap exceeds the bound.
Method, guards, and what is not measured here: METHOD.md.
7 arms, zero invalid.
Verdict
| Does the derivation run in a real compiled Sky app? | Yes. Every arm's [sky.gc] banner is read back from the running process — GOMEMLIMIT=9.9GB derived from 16.0GB of machine memory less the OS and the embedded cluster's share; GOGC=400. |
| Does the bound hold at n = 100/300/500? | Yes. 432 / 650 / 973 MB against a 1,044 MB limit. RSS lands under the limit at every level. |
| Does the measured workload fit an e2-small? | Yes, at 59% with margin. See the table below. |
| Does exceeding the bound kill the app, or slow it? | Slow it. Under an unsatisfiable 201 MB bound the app served all 300 sessions at error_rate 0 and patch_rate 1.0, at 50% throughput. No abort, no OOM, no death spiral. |
Does it fit an e2-small?
RSS here is macOS/arm64; divided by 1.17 for a Linux estimate, a factor
borrowed from gcp-x86-capacity-20260816 and not re-measured here. The
embedded cluster's peak is measured in the same arm. OS taken at 256 MiB.
| arm | app (est. Linux) | + PostgreSQL | + OS | total of 1,977 MB |
|---|---|---|---|---|
e2small-n100 | 369 MB | 80 MB | 256 MB | 706 MB — 36% |
e2small-n300 | 556 MB | 80 MB | 256 MB | 892 MB — 45% |
e2small-n500 | 832 MB | 80 MB | 256 MB | 1,168 MB — 59% |
That table is generated by harness/analyse.sh from the arms' own acct.txt;
no number in it was transcribed by hand.
The 41% that is left is the point. GOMEMLIMIT is a soft limit, so the
margin is not slack — it is what the process overshoots into when a workload
genuinely needs more, instead of being OOM-killed. The overbound arm below
measures exactly how much of it gets used.
The arms
| tag | n | GOGC | GOMEMLIMIT | peak RSS | int/s | estab | err | load1 |
|---|---|---|---|---|---|---|---|---|
default-n100 | 100 | derived | derived (9.9 GB) | 419 MB | 2,471 | 100/100 | 0 | 10.28 |
default-n300 | 300 | derived | derived (9.9 GB) | 893 MB | 2,647 | 300/300 | 0 | 11.44 |
default-n500 | 500 | derived | derived (9.9 GB) | 1,338 MB | 1,663 | 500/500 | 0 | 15.82 |
e2small-n100 | 100 | 400 | 996 MiB | 432 MB | 3,040 | 100/100 | 0 | 11.15 |
e2small-n300 | 300 | 400 | 996 MiB | 650 MB | 3,432 | 300/300 | 0 | 10.26 |
e2small-n500 | 500 | 400 | 996 MiB | 973 MB | 2,454 | 500/500 | 0 | 12.94 |
overbound-n300 | 300 | 400 | 192 MiB | 218 MB | 1,727 | 300/300 | 0 | 16.14 |
default-* is the shipped binary with nothing in the environment; on this
16 GB host the derived bound is ~9.9 GB and does not bind, so those arms are the
GOGC=400-unbounded case. e2small-* supplies the exact figures the rule
derives for a 1.93 GiB instance under --embed.
default-n500is contaminated and is not used for any comparison. It ran atload1 15.82and the app got only 164 s of CPU against ~240–290 s in every other arm — a starved app queues, and a queue is live heap. Its 1,338 MB and its 1,663 int/s are artefacts of two sibling agents' benchmarks, not of the collector. The clean unbounded control used below is the parent run's, on the same host, app and store.
What the bound does, against a clean control
The parent run's unbounded GOGC=400 arms at n=500 are the right comparison,
because they were taken on a quiet host:
| config | limit | peak RSS | source |
|---|---|---|---|
GOGC=400, unbounded | — | 1,016 / 1,170 MB | gogc-postgres-20260816, n500-gogc400-b1/b2 |
GOGC=400 + 750 MiB | 786 MB | 759 MB | same run, combo-n500-gogc400-750MiB |
GOGC=400 + 996 MiB (this rule, e2-small) | 1,044 MB | 973 MB | this run, e2small-n500 |
The three lines say one thing: RSS tracks the bound and settles just under it, and below the bound the multiplier governs. A larger bound is not wasted memory, it is headroom for session count — which is why the rule scales the limit with the machine rather than pinning the 750 MiB that was measured.
The falsifier, closed
gogc-postgres-20260816 listed as its first falsifier:
A workload whose live heap legitimately exceeds the derived limit. Every arm here stayed under its bound, so the thrash case — collector running continuously against a limit it cannot satisfy — is untested. It is the main risk of shipping a limit.
overbound-n300 tests it: a 192 MiB limit, below the app's own working set
at 300 sessions, so the collector cannot reach its target however hard it runs.
| Peak RSS | 218 MB against a 201 MB limit — the runtime EXCEEDED the soft limit rather than failing an allocation |
| Sessions established | 300 of 300 |
error_rate / patch_rate | 0 / 1.0 — every interaction correct |
| Throughput | 1,727 int/s against 3,432 for the same n=300 with a workable bound — 50.3% |
| Aborted? | No. The RSS watchdog never fired; the generator returned 0 |
The failure mode is degraded throughput, and the degradation is the one Go
documents. The runtime's GC CPU limiter caps the collector at 50% of CPU
precisely so that an unsatisfiable GOMEMLIMIT cannot become a death spiral,
and the measured 50.3% of throughput is that cap showing through. Correctness is
untouched: not one interaction failed.
That is why GOGC stays a multiplier and why gcMinMemoryLimitBytes exists —
the runtime declines to set a bound at all below 256 MiB, which is ~1.8× the
138–145 MB the stock collector already peaks at. The unit test
TestExceedingTheLimitDegradesThroughputRatherThanDeadlocking asserts the same
property in-process.
What is still not measured
Named, not assumed:
- x86/Linux. Every arm is arm64/macOS. The ÷1.17 adjustment is borrowed.
- A real e2-small. The bound is supplied through the environment at the
value the rule derives; what is simulated is the machine, not the setting. The
RAM → limit arithmetic is covered instead by
gc_tuning_test.goover fourteen machine sizes, including the cgroup path a container takes. - Throughput. This host could not resolve it today (
METHOD.md). The int/s column is reported for completeness and relied on only where the comparison is within an arm pair taken minutes apart at similar load — the overbound 50.3%, whose predicted value was known in advance. - The overbound case at n = 500. It was queued and abandoned unstarted: it needed a quiet host and the host was needed for the verification gates. The n=300 arm closes the falsifier on its own — the property under test is qualitative (degrades vs dies) and it degraded, at the exact fraction Go's CPU limiter predicts. An n=500 arm would have measured how much further it degrades, which no decision here depends on.
- n > 500, multi-replica, and serverless. The serverless branch takes the
bound and declines the multiplier by construction; that decision is argued in
runtime-go/rt/gc_tuning.goand gated by test, not measured here. GOGCbetween 200 and 800. Unchanged from the parent run: 400 beats 200 and 800 on the memory/throughput trade at these session counts; it is not established as the optimum.
Layout
METHOD.md what is being verified, the guards, and the conditions
results.tsv one row per arm, with the analysis-time validity verdict
runs/<tag>/ per-arm acct.txt, 1 Hz rss.tsv, app.log (with the [sky.gc] banner)
harness/ runone.sh, sweep.sh, analyse.sh, and both mutation provers