Amortizing the per-token host-sync setup tax in fused all-layer q4 decode on Apple M1 Ultra (1.57x), then occupancy-bound

1. Contribution

Claimed. A session-keyed setup cache that amortizes the per-token host-side setup tax in a fused all-layer q4 decode path on a single Apple M1 Ultra (48-core GPU, 800 GB/s). On a q4 30B-class MoE single-stream decode benchmark, this raises measured steady throughput from 25.07 to 39.25 tok/s (1.57x), byte-identical greedy output, sovereign (pure-Metal, no MPS, no CPU BLAS in decode or prefill). We also report the campaign terminus: with the setup tax removed, single-stream decode is occupancy-bound (small-M int4 GEMVs at well under peak bandwidth), so host-sync is the last amortizable bubble, not the root lever.

NOT claimed. We do NOT claim 39.25 tok/s is the production-serving rate (see §3 and §5 for a separately reported, lower production-serving figure measured under different conditions). We do NOT claim a GPU-compute speedup — the genuine GPU-compute (wait) phase is the M=1 arithmetic-intensity floor and is not amortized. We do NOT claim multi-run reproduction or variance bounds (single measured run). We do NOT claim the win transfers to the batched/aggregate-throughput path beyond the parity-safe port noted below.

2. Methods

3. Results

Profile-first decomposition of the per-token decode wall (real q4 30B-class MoE, single-encoder path), baseline steady over roughly 39 tokens, approximately:

• wall ~36.95 ms; setup ~10.65 ms (~28.8%); genuine GPU-compute wait ~26.23 ms (~71.0%); commit/tail negligible.

The setup phase rebuilt, every token, the per-layer-per-expert-per-projection bank offset tables (on the order of tens of thousands of locked dictionary lookups) and re-allocated plus host-copied the per-layer norm/offset buffers — all byte-identical token to token for a stable decode session. The GPU-compute wait is the M=1 floor and is NOT amortizable (token N+1 depends on token N's argmax, so there is no overlap).

Fix (session-keyed setup cache, mirroring an existing KV-pool session cache): on a hit, offset tables are reused and per-layer host buffers are persistent rather than borrowed from a recycled pool; they are rebuilt only on session/geometry change. Cached steady, approximately:

• wall ~22.65 ms; setup ~0.90 ms (~4.0%).

Setup falls roughly 10.65 to 0.90 ms (about -9.75 ms; ~28.8% to ~4.0%). The GPU-compute wait also fell (~26.23 to ~21.7 ms) — freeing the CPU from the locked lookups lets the GPU be fed earlier. Steady throughput 25.07 to 39.25 tok/s = 1.57x.

Gates (all PASS): byte-parity (no divergence, 2 prompts/lengths); 1.57x throughput; no CPU BLAS in decode/prefill; memory-safe (sole q4 ~15.4 GB resident, ample free memory).

Reconciliation vs a separately published production-serving figure. These are different conditions and neither supersedes the other:

• 39.25 tok/s = host-sync amortization measured on the isolated decode benchmark path (steady, cold first token excluded, single-encoder, setup cache ON).
• A lower (~24 tok/s) figure = a separately published production-serving steady single-stream number over the full serving path, which under its own serving fixes requires a GPU top-k path default-ON for the single-encoder to fire; the isolated benchmark with no such flag reports ~25.3 steady.

The gap is attributable to condition differences (isolation benchmark vs full serving path; flag/warmup state), not to a contradiction. We report both with their conditions and do NOT assert either as the canonical single-stream number.

4. Ablations / negative controls

• Cache OFF vs ON (the amortization itself). OFF = baseline 25.07 tok/s with setup at ~28.8% of wall; ON = 39.25 tok/s with setup at ~4.0%. Byte-identical output both ways isolates the win to host overhead, not math.
• Campaign envelope negatives (context for the occupancy-bound claim), from prior waves: barrier-bound FALSE; occupancy-reshape-fixable FALSE (M=1 arithmetic-intensity wall, scalar kernel already at floor); batch-amortizes-bandwidth FALSE (aggregate sub-linear, saturates around ~35.6); tiling-folds-weight-reads FALSE (1 of 24 shape/batch configs won, MoE never crossed, compute/occupancy-bound at well under peak BW). These are reported in their own waves; here they establish only that host-sync was the last amortizable bubble.

5. Threats to validity / limitations

• Single run, no variance. Source records one measured steady run per condition; no standard deviation, no independent reproduction. Evidence level is therefore experimental.
• Benchmark path, not serving path. 39.25 tok/s is the isolated decode benchmark number. It is NOT the same measurement as the separately reported ~24 tok/s production-serving steady. Do not quote 39 as the serving rate.
• Steady excludes warmup. The cold first token is excluded; short generations that include warmup will report lower.
• Hardware-specific. Measured only on one M1 Ultra (800 GB/s). The 1.57x and the ~10.65 ms setup tax are properties of this geometry and host overhead; they do not transfer to other chips or model shapes unchanged.
• GPU compute is not improved. The wait phase is the M=1 occupancy floor (well under peak bandwidth utilization); the win is purely reclaimed host idle. Going below ~22.65 ms/token requires attacking compute (batched/grouped-expert GEMM, fp8) or algorithmic speculative decode — both out of scope here and separately found not to lift single-stream decode.
• Persistent-buffer cost. The cached path holds per-layer host buffers persistently (a handful of tiny buffers per session) in exchange for the per-token tax; negligible vs the ~15.4 GB resident, but a real allocation-lifetime change.
• Batched-path transfer — parity-safe port only, NOT a merged serving feature. Only a parity-safe port of the setup cache to the batched single-encoder path is recorded (B=1 +37%); the batched aggregate remained sub-linear (~35.6) and is not claimed to benefit beyond that. It was not promoted to a standalone serving feature.

6. Reproducibility

• Mechanism. Single-stream host-sync change: a session-keyed setup cache for the fused all-layer decode path, mirroring an existing KV-pool session cache; offset tables and per-layer host buffers persist across tokens and rebuild only on session/geometry change. A parity-safe port of the same cache to the batched single-encoder path was recorded (parity-only, not merged as a serving feature).
• Scope of change. Localized edits to the fused decode kernel implementation plus the isolated decode benchmark driver.
• Configuration. Host-sync per-phase profiler and a setup-cache toggle (default ON after parity proven). Sole q4 packed-int4 resident (~15.4 GB), never bf16; VM-gated, ample free memory.
• Artifacts. Per-condition results plus profile and cached split records, recomputable off-disk.
• Build. Recent verified build; sovereign, no CPU BLAS in decode/prefill.
• Not recorded in source. Exact toolchain version, OS build, and any run-to-run variance.