Batched resident MoE experts on Apple Silicon: a flat parallel expert pass, ~1.6x decode, parity bit-identical (single-run)

1. Contribution

Claimed. A single engineering change to a text-diffusion Gemma MoE (26B-A4B-class) forward on Apple Silicon that collapses the nested per-(token,expert) expert FFN path into one flat parallel pass per layer, plus the measured effect of that change on a single decode run:

• The prior path looped over tokens, then over the top-8 routed experts, and each tiny M=1 expert GEMV spawned its own narrow parallel launch. Between those launches the CPU cores were starved.
• The new path builds an immutable (token, expert, weight) job list for the whole layer, pre-fetches the F16-resident expert slices sequentially (read-only afterwards), then runs one flat parallel pass over all jobs (gate/up projection → gated activation → weighted down projection per job).
• On one decode run (small canvas, 4 steps) this cut decode wall by ~1.6x, with CPU utilization rising toward core saturation, while producing bit-identical decode token ids and an unchanged forward argmax.

NOT claimed. This is not a benchmark report and not a throughput claim about autoregressive MoE serving. It does not claim multi-run statistics, a tokens-per-second figure, GPU expert-kernel batching, KV-cache reuse, any improvement to decode answer quality, or applicability to an autoregressive MoE serving path. It is a single-machine, single-run result. It is distinct from per-request-stream KV-isolation and batched-concurrent serving work — those parallelize across request streams; this parallelizes the expert FFN inside one single-stream decode.

2. Methods

3. Results

All numbers are as reported in the engineering record. Single run; no variance recorded.

• The ~3x cumulative figure is not attributable to this change alone — the record states it is the batched-experts change combined with a prior output-projection batching (folding W separate M=1 calls into one M=W GEMM). This change's own contribution is the ~1.6x.
• Parity: decode produces the same canvas token ids as the scalar path (decoding to a short natural-language continuation); the forward argmax is unchanged. Reported as bit-identical by construction (same F16 weights, same per-token top-k summation order).

4. Ablations / negative controls

No formal ablation or negative control accompanies this change. The implicit comparison is the prior scalar/nested path in the same binary (the "before" column). Bit-identical output is the correctness anchor; no separate parity harness run is recorded specifically for this change beyond the canvas-id and argmax checks above.

5. Threats to validity / limitations

• Single run, no variance. One small-canvas/4-step decode; no repeats, no confidence interval, no warm/cold distinction recorded. Hence the evidence level is experimental, not reproduced.
• No tokens-per-second and no throughput claim. The numbers are wall-clock for one configuration; nothing about aggregate or batched serving throughput is implied.
• CPU-bound expert pass. The speedup comes from saturating CPU cores, not from a GPU expert kernel. Gains are tied to this 20-core-class machine and will differ on other chips; CPU utilization did not reach the full core ceiling, so headroom and scaling are unquantified.
• Configuration sensitivity. Measured only at small canvas, 4 steps. Larger canvases, more steps, and longer prompts are not measured; the job-list count (and thus parallel efficiency) scales with n_tokens × top-8, which was small here.
• Quality is unchanged, not improved. A downstream regression suite on the same decoder reports a partial verdict — answer-hit roughly 27% versus a sampled-denoiser oracle near 96%, a structural argmax-denoiser limitation. Bit-identical means this speedup neither causes nor fixes that gap; it does not make the decoder production-ready.
• Hardware/build under-specified. Exact chip, OS, compiler/toolchain version, and build flags are not recorded.

6. Reproducibility

• Scope: a localized perf change to the expert FFN path of the diffusion-Gemma forward, plus a documentation record. Single-author engineering record dated 2026-06-15.
• Model: text-diffusion Gemma MoE, 26B-A4B-class, 4-bit quantized.
• How to reproduce (from sources): run the diffusion-Gemma decode at small canvas, 4 steps on the same Apple Silicon machine, before vs after the change, comparing decode wall, CPU%, decode canvas token ids, and forward argmax.
• Not recorded in source: exact CLI invocation/flags, seed, prompt for the timing run, environment (OS/compiler version), and any multi-run timing distribution.