Single-stream int4 decode of a 30B MoE on Apple Silicon: steady-state throughput with byte-identical, vendor-BLAS-free parity

1. Contribution

Claimed. On a single Apple-Silicon machine, serving a 30B sparse-MoE in packed-int4 (roughly 15.4 GB resident), single-stream greedy decode reaches a steady-state throughput about 1.87x higher than a prior per-token execution path. The improvement comes from two changes: (a) folding all decode tokens into a single GPU command buffer, collapsing the per-token internal synchronizations into a single synchronization point, together with performing the key/value cache write on-GPU rather than on the host; and (b) root-causing a silent roughly 2x serving regression to an optimization gate that shipped disabled by default. The optimized path stays greedy byte-identical to a reference forward pass (cosine approximately 1.0) and uses zero vendor BLAS calls.

NOT claimed. This is an engineering / correctness-validation result, not a novel algorithm and not a benchmark-scale evaluation. It does NOT claim generation quality (byte-identical parity is numerical parity to a reference forward pass, not an evaluation of output quality). It does NOT claim competitiveness with a popular external single-stream runtime (measured roughly 2.2x ahead). It does NOT claim linear multi-stream scaling (measured roughly 1.4x at batch 4). It does NOT claim anything for the bf16 path, sampled (non-greedy) decode, or non-Apple-Silicon hardware. Throughput figures are single-run, single-box, with no reported variance.

2. Methods

3. Results

4. Ablations / Negative controls

This is a root-cause toggle, not a controlled negative-control sweep; only the two states above are recorded.

5. Threats to validity / Limitations

• Parity is not quality. Cosine approximately 1.0 and greedy byte-identical confirm the optimized path reproduces the reference forward pass numerically. They say nothing about whether the model's outputs are good.
• Sample size / no benchmark scale. Parity rests on long- and short-prompt real-shard runs (exact prompt count not recorded); this is a correctness smoke test, not a benchmark-scale evaluation. Throughput figures are single-run, single-box, with no reported variance or confidence interval.
• Warmup-sensitive headline. The quotable steady-state figure explicitly excludes a few warmup tokens; short generations dominated by warmup average roughly half that. Quoting the steady-state number for short generations would overstate.
• Baseline-comparison weakness. The external-runtime comparison (and the resulting roughly 2.2x gap) carries no recorded version, build, model-quantization match, or run conditions; treat it as directional.
• Concurrency caveat. Single-stream is serialized today; batch-4 aggregate is only about 1.4x (a concurrency reversal, honestly anchored against an earlier verifier's inflated figure). No linear scaling is demonstrated.
• Precision/scope ceiling. Results are bound to packed-int4, greedy decode, on the measured Apple-Silicon machine. bf16, sampled decode, and other hardware are out of scope.
• Evidence level. This rests on a small parity set plus single-run throughput, so "reproduced" should be read as "parity re-confirmed across the fix," not "independently re-benchmarked at scale."

6. Reproducibility / Provenance

• Change set: two structural changes — (1) single-encoder decode folding all tokens into one GPU command buffer with an on-GPU key/value cache write, and (2) enabling the previously default-disabled optimization gate. Exact build identifiers and commands are withheld.
• Version pins: SoC/RAM, toolchain, and external-runtime versions are not recorded in source notes.
• Validate-the-oracle caveat: parity is asserted against a reference forward pass on real shards; the reference's own correctness should be confirmed before trusting the cosine figures.
• Sovereignty check: zero vendor BLAS calls (platform numerics only, no third-party ML framework) on the parity runs; the int4 path dequantizes on-chip from packed int4, keeping the 30B model resident in roughly 15.4 GB rather than roughly 60 GB at bf16.