Sovereign gpt-oss-20B (MXFP4) on Apple Silicon: a fine-grained-MoE small-matrix occupancy ceiling

1. Contribution

Claimed. A sovereign (no vendor BLAS) Apple-Silicon forward port of gpt-oss-20B in its native MXFP4 weight format, with two genuinely new kernels — an MXFP4 dequant-GEMV and learned per-head attention sinks — verified for correctness against a Hugging Face golden, plus a single-machine performance campaign that lands and then closes out at a measured occupancy ceiling. The honest negative is a first-class result: after a proven single-command-buffer fusion and a proven multi-SIMD cooperative-GEMV occupancy lever, single-stream decode tops out near ~14.6 tok/s, below the pre-registered 18 tok/s continue threshold, because fine-grained-MoE expert matrices are too small to fill the GPU. A microbench on real gpt-oss-120B expert weights shows the 120B inherits the identical ceiling (experts are the same shape as 20B), so 120B was not ported.

NOT claimed. No claim that gpt-oss-20B/120B is competitive with mature external runtimes (which report far higher single-stream tok/s on this silicon class). No claim the ~36/27 tok/s prediction from a prior deep-dive held — it did NOT (see §5). No claim of a multi-run reproduced tok/s figure — every tok/s here is a single measured session. No claim about 120B end-to-end decode (only a single-expert microbench was run; the full model was never loaded). No production-serving claim.

2. Methods

3. Results

Correctness (vs Hugging Face golden, real shard).

• MXFP4 reader: weight-level cross-validation on a layer-0 expert, 474048 elements, max_abs = 0.0 (BIT-EXACT); transpose orientation verified. The same MXFP4 reader is bit-exact (max_abs=0) independently across silicon, CUDA, and ROCm.
• Per-component layer-0 cosine: embed 1.0, attention(+sinks) 0.99998, MoE(MXFP4) 0.99999, layer-0 output 0.99999.
• Full 24-layer final-logits: min cosine 0.9969, argmax matches (" Paris"); coherent/factual decode.
• Sliding-window seq>128 (140 tok): argmax exactly matches the oracle (201088-way). Full-stack 140-tok cosine 0.99830 is the f32-vs-bf16 24-layer × 140-token accumulation envelope, not a sliding bug.
• YaRN reproduced natively: inv_freq max_abs 3.7e-9, attention_scaling delta 9.5e-10.

Throughput (single-stream, KV-cached decode, M1 Ultra, same-session steady).

After both proven levers, effective bandwidth was only ~7% of M-series peak — the decode is dispatch/latency-bound (49 CB/tok + 4 sequential experts × 2 small-block GEMVs), NOT GEMV-bandwidth-bound.

Pre-registered gate outcome. Thresholds were 24+/30+/<18 tok/s. Measured 14.6 < 18 → CLOSE OUT.

120B viability microbench (NOT a port). The shipped cooperative dequant-GEMV (16-byte coalesced vector load + on-chip dequant) was run on real 120B layer-0 expert-0 MXFP4 weights, sliced by exact safetensors byte-offset from one shard (1 of 128 experts; the full model was never loaded).

Confirmed by measurement: 120B per-expert matrices are the same shape as 20B (hidden/intermediate 2880, top-4); 120B's larger active footprint is more experts (128 vs 32) + more layers (36 vs 24), not bigger experts. → 120B inherits the identical small-matrix ceiling; same-or-worse (more layers). CLOSE OUT.

4. Ablations / negative controls

• kill-sinks (attention sinks removed): layer-0 attention cosine collapses to 0.548 and decode degenerates to prompt-echo → attention sinks are load-bearing.
• kill-sliding at T<128: null effect (honest — the sliding window only differs from full attention once sequence length exceeds the 128-token window).
• MXFP4 scalar-load negative control (byte-by-byte instead of vector load): stuck at 6.1% / 50 GB/s; the aligned vector path lifts the load ceiling to 95% → the 16-byte alignment gain is real and the format itself is not the wall.
• The structural ceiling is NOT a kernel defect: the cooperative-GEMV math is identical to the legacy kernel (only the cross-block fp32 sum is re-associated), and the result lands on the same 20.2% line for both sizes.

5. Threats to validity / limitations

• Single run, no variance. Every tok/s number is one measured session; no multi-run reproduction or variance interval is recorded in source. Evidence = experimental, not reproduced.
• The prior deep-dive prediction was wrong, by design honesty. An earlier read-only deep-dive predicted ~36 tok/s (20B) / ~27 tok/s (120B) from a 16.6% effective-BW model. The actual measured ceiling is ~14.6 tok/s. The prediction assumed the campaign would reach the bandwidth-bound regime; it instead revealed a small-matrix occupancy/dispatch ceiling. This report supersedes that projection — do not cite ~36/27 as achieved.
• 120B is unmeasured end-to-end. Only a single-expert GEMV microbench ran; the full model was never loaded, no 120B decode tok/s exists. The 120B conclusion rests on a config-shape argument plus a per-kernel microbench, not a port.
• One prompt family / short generation. Correctness gated on "The capital of France is" plus a few short continuations; tok/s on 4 prompts × 32 tokens. No long-context throughput or large-batch numbers.
• Oracle precision floor. Min cosine 0.9969 is the f32-engine vs bf16-oracle accumulation envelope over 24 layers; this is an envelope, not a guaranteed bound, and the engine has no f32 oracle of its own.
• External-runtime ceilings are third-party, not reproduced here.
• Untried lever. Grouped-expert GEMM batching (K=4 active experts in one GEMM) was identified but not implemented; the ceiling claim is for the levers actually run (CB fusion + cooperative GEMV), not a proof that no kernel could ever exceed 18.
• 120B threshold nuance. The cooperative kernel sits literally on the 20% line, so a strict ">20%" reading technically fires — but it is the same number 20B already closed out at 14.6 tok/s end-to-end; the gate's intent ("clearly higher than 20B") is false.

6. Reproducibility (high-level)

• Path. An isolated standalone forward executable on the silicon text engine, with a dedicated fused-decode CLI mode. A cooperative SIMD-group count is configurable via an environment variable (default 4; the MXFP4 path is capped at 2 by the small-block matrices).
• Inputs. M1 Ultra; a recent Hugging Face transformers oracle; real gpt-oss-20B/120B shards. Exact OS and toolchain versions for the tok/s sessions were not recorded in source.
• Caveat. Per-tok/s figures are single sessions without recorded variance (see §5).