Does Sky.Live's throughput scale with cores?

Yes — very nearly linearly with physical cores. The observation that prompted this run (1.0 dedicated core → 299 int/s, e2-medium's 1.85 app-cores → 345 int/s: ~1.85× the CPU for ~1.15× the throughput) does not survive a same-box test. It was a cross-instance-family artefact, and the agent that flagged it as indicative rather than conclusive was right to.

On one machine, one binary, one generator, one session count, arms counterbalanced across three blocks:

Ranges are ≤ 3.3% of the median at every level and the three blocks agree to within 2% arm-for-arm, so the curve is not a noise artefact.

Verdict

Roughly linear to 4 threads; the 4 → 8 flattening is hyperthreading, not the application. Confidence: high, for the reason below — the flattening was reproduced under CPU pinning, where the number of threads is held constant and only the physical cores they sit on changes.

e2-standard-8 does not have 8 cores. lscpu reports 4 cores per socket, 2 threads per core; thread_siblings_list pairs cpu0/4, 1/5, 2/6, 3/7. So GOMAXPROCS 1 → 4 walks up four distinct physical cores, and 4 → 8 adds their SMT siblings.

Same GOMAXPROCS = 4, same load, same binary, only the affinity mask changed:

Four threads on two physical cores deliver 70% of four threads on four physical cores. Against the unpinned GOMAXPROCS=2 figure (865.5, which Linux spreads across two distinct cores), the second SMT thread on each core is worth 1.27× — and the measured 4 → 8 step is 1.17×. That is what SMT does; it is not a scaling defect.

Per physical core, throughput scales at 79–80% efficiency per doubling (1.00 → 1.79 → 3.18).

What this means for the target

A larger instance is a legitimate route, and the sizing guidance should count physical cores, not vCPUs. Any capacity number derived from a GCE vCPU count overstates the machine by roughly the SMT factor. AGENTS.md's instance table should say so.

Per-interaction cost is not the only lever — but it remains the best one, because it compounds with cores and because two cheap levers were measured here that do not require a bigger machine at all (below).

If it is sub-linear, why — the named causes, with evidence

Three separate instruments were run. They disagree with each other in a way that is itself the finding, so all three are reported.

1. A mutex profile at GOMAXPROCS=8 names four locks — and they are real

runtime.SetMutexProfileFraction(1), sampled as a delta across the measurement window (both profiles are cumulative from process start, so the start reading is subtracted; what is reported is contention during the load, not contention plus startup). Full output in profiles/g8.txt.

The same profile at GOMAXPROCS=1 totals 39.97 microseconds — a 14-million-fold difference, i.e. the contention is entirely a parallelism effect, as expected.

Std.Ui.Lazy's LRU does not appear at all, at any level. Checked before blaming it, as instructed: examples/19-skyforum contains no lazy call, so the 1024-cap LRU behind lazyCacheMutex (lazy.go:48) is not on this app's path. It may still matter for an app that uses Ui.lazy; this run says nothing about that.

Contention on a single session is ruled out. Throughput is nearly flat in session count — at GOMAXPROCS=8: n=25 → 1,704.9, n=100 → 1,796.2, n=400 → 1,898.0. The sweep drove 100 distinct sessions and was never serialising on one sess.mu.

2. …but lock waiting is not where the CPU goes

Contended locks show up in CPU as lock2 / futex / procyield. Summed:

Meanwhile the CPU cost of an interaction rises from 1,876 µs to 3,803 µs (2.03×) between 1 and 8 threads, and it rises across every path at once — reflect.Value.call 1,185 → 2,134 µs/int, mallocgc 624 → 1,091, systemstack 547 → 1,361, gcBgMarkWorker 245 → 473. That uniform inflation is a memory-system and collector signature, not one lock.

3. GC is the largest single lever measured, and it is level-independent

GOGC sweep, same everything else:

app_cores is unchanged across the GOGC arms at GOMAXPROCS=8 (6.79 / 6.74 / 6.81) — the same CPU does a third more work.

The Go runtime's own gctrace reports GC at 8% of CPU at both GOMAXPROCS=1 and 8, so the 33% is not GC's direct CPU. It is the collector's duty cycle: at GOMAXPROCS=8 the app runs 2,092 GC cycles in 92 s (23/s) against 750 at GOMAXPROCS=1, with a mark phase of 11–12 ms per ~26 ms cycle — the write barrier is on for roughly 45% of wall time. Relaxing the pacer removes barrier duty, not collector CPU.

The gain is the same 1.33–1.37× at 1, 4 and 8 threads. GC is therefore a per-interaction cost lever, not a parallel-scaling limiter. It does not explain the shape of the curve; it shifts the whole curve up.

4. Sharding into processes buys 33% — and GOGC buys the same 33%

The direct test of whether the ceiling is inside one address space. Same box, same 8 hyperthreads, same 100 total sessions, same binary; only the number of processes changes, so every process-wide lock, the Go heap and the GC pacer are duplicated rather than shared.

Eight independent single-threaded processes reach 4.92× the single single-threaded process — against a hardware ceiling of about 5× (4 physical cores × the measured 1.27 SMT factor). Shared-nothing extracts essentially all of the hardware.

But note the two numbers side by side:

8 processes × 1 thread = 2,384 int/s. One process × 8 threads with GOGC=800 = 2,404 int/s.

One environment variable buys what the whole sharded topology buys. The intra-process penalty is dominated by the shared heap and its collector, not by the four locks the mutex profile names — which is why removing the locks (sharding) and removing the barrier duty (GOGC) land on the same number.

For the shared-nothing proposal specifically

The coordinator framed this as "if a lock dominates, fix the lock; if it is GC, sharding cannot help". The measurement does not split that way, so here it is plainly:

• Locks are genuinely contended — memMu, sessionLocker.mu and the goroutine→session sync.Map are process-wide and are each hit on every interaction. That is a real defect and worth fixing on its own merits.
• They are not what caps throughput. Lock spin is 4.7% of CPU at 8 threads; fixing all four cannot plausibly return 33%.
• Sharding does help — by 33% — which contradicts "if it is GC, sharding cannot help". The mechanism is that separate processes have separate heaps and separate pacers, not that they dodge the locks.
• The same 33% is available for one env var, without an architecture. So the shared-nothing rebuild is not justified by this data as the route to that 33%. Reducing allocation per interaction earns the same thing permanently and compounds with cores.

On the DB-pool half: this run cannot speak to it, and does not contradict the x86 finding. Every arm here ran SKY_LIVE_STORE=memory, deliberately, to isolate application CPU. There is no PostgreSQL in this corpus at all, so backends_max 7–8 against max_connections=56 stands unchallenged. Do not read this run as support for or against per-core DB pools.

Was the generator the bottleneck? No — shown, not asserted

Established three ways, at every point:

1. The generator ran on its own 8-vCPU box (skygmp-gen, same zone, internal IP, ~0.2 ms hop), so it never competed for the app's cores — which matters precisely at GOMAXPROCS=8, where sharing a host would have manufactured the sub-linear result under test.
2. skyliveload's own getrusage accounting: 2.2% of its machine at GOMAXPROCS=1 rising to only 6.1% at GOMAXPROCS=8; its generator_possibly_saturated flag (trips at 70%) is false in all 47 recorded runs.
3. The generator box's /proc/stat busy fraction, measured independently across exactly the load window: 1.9% → 6.9%, peaking at 9.9% in the 8-process arm.

The generator had >90% headroom at every point on the curve.

Method guards, and the two defects they caught

Carried from the capacity harness and extended:

• -hid-context '>▲<' names the handler by rendered text; -setup scripts the sign-in the vote handler needs; -min-patch-rate 0.9 refuses a run that produced no patches; a four-interaction -self-check runs as a precondition before every measurement window. All runs report valid: true, patch_rate 1.0, 2 patches per interaction, error rate 0.
• The app's session store is asserted from its own startup banner, and GOMAXPROCS is read back from /proc/<pid>/environ — an arm that silently ran at the wrong level would have produced exactly the flat curve this run exists to test for.
• The app pid comes from pgrep -x app cross-checked against the :8000 listener pid, per the prior run's pgrep -f defect.
• /proc/stat tick rate was measured, not owed: 800 jiffies/s against 8 × 100 owed, so CPU attribution from inside this guest is sound — unlike the 65–94 Hz seen on the small shared instances of the capacity run.

Defect 1 — a leftover binary served an arm. The first two-process attempt asserted expected 2 app processes, found 1 and was discarded. Cause: pkill -x app does not match app-prof, so the instrumented GOMAXPROCS=1 binary left over from the profiling arm still held :8000, and that arm's two ports read 415 and 1,305 int/s. Without the pid-count assertion this would have been recorded as a valid, patch-bearing, plausible 1,721 int/s. Fixed by killing both executable names; the discarded attempt is not in results.tsv.

Defect 2 — zsh noclobber served me a stale file for three cycles. Two profiling batches appeared to run and report identical throughput to 13 decimal places. > had silently refused to overwrite an existing prof.log (file exists:) so the block never ran and I was re-reading the previous batch. The first batch had also produced no profiles at all, because remote_prof.sh was written after provisioning and never uploaded. Both fixed; harness scripts use >| throughout.

A window-selection defect was caught before it mattered. The sampler outlives the load by ~45 s so a slow teardown cannot truncate it, which made a tail -N window average the app's idle tail into its CPU — the trial read app_cores 0.747 at GOMAXPROCS=1 where the raw jiffy slope over the load was 1.05. The window is now selected from the trace itself (rows at the run's plateau connection count), and app_cores is recomputed at analysis time rather than trusted from the driver.

Conditions

Not measured — named, not assumed

• PostgreSQL. Every arm used the memory store. Nothing here confirms or contradicts the capacity run's backends_max 7–8 / 0.08–0.12 pg-cores.
• Whether fixing the four named locks would change the curve. The mandate was to measure, not implement. Their 4.7%-of-CPU spin cost bounds the available gain from below, but a counterfactual was not run.
• More than 4 physical cores. e2-standard-8 is 4 cores; whether the 79–80% per-doubling efficiency holds at 8 and 16 physical cores is untested. It is the obvious next sweep and needs an n2/c3 instance, not an e2.
• Std.Ui.Lazy's LRU under an app that uses it. Absent from this app's path, so unexercised.
• Any arm on the M1. Everything here is x86; the existing corpus's M1 : x86 factor of ~2.96× is unchallenged and unre-measured.
• The absolute numbers against the capacity run's Xeon. This box is an AMD EPYC 7B12; its single dedicated thread reads 484 int/s where the capacity run's Xeon read 299. That is a CPU difference and is not a correction to the Xeon figure. The scaling ratios, which are all within-box, do not depend on it.

Layout

results.tsv         the 12-run counterbalanced sweep, one row per run
sweep.log           its driver output
pin.log             the SMT affinity experiment
shard2.log          2 processes x 4 threads
shardk.log          4x2, 8x1, and the 1x8 control
diag.log            GOGC sweep at 8, session sensitivity, gctrace arms
gogclvl.log         GOGC=100 vs 800 at GOMAXPROCS 1 and 4
prof.log            the profiling arms
profiles/g{1,4,8}.txt   mutex delta, block delta, CPU flat + cum, rendered
                        against the instrumented binary (98 MB, not committed)
runs/<tag>/         per-run load.json, idle assertion, 1 Hz sample, app.log
harness/            every script, including zz_gmpprobe.go