Apple-Silicon-friendly LLM architecture: substrate laws reverse-engineered from a model bake-off

1. Contribution

Claimed. A reverse-engineered spec — five design rules — for what makes an LLM architecture run fast under single-stream 4-bit decode on one Apple M1 Ultra in a custom Metal decode engine (no MPS, no vendor BLAS). The rules are derived bottom-up from a model bake-off: five structurally distinct families were each taken to a landed port or a decisive viability gate, profiled on real GPU timestamps, and recorded at the exact wall where each broke. The rules are: (1) low active parameters per token; (2) vector-load-aligned quantization format; (3) expert matrices large enough to fill the GPU at batch size one (or batched inside a grouped-GEMM, single-encoder runtime); (4) uniform attention geometry; (5) fusion-friendly, non-hybrid weight layout. Each rule is paired with its hardware reason and the candidate that proved it by violating it. The meta-finding is the headline contribution: architecture identity does not predict runtime cost — two models with identical spec sheets can diverge two-to-threefold in achievable tok/s on weight layout, activation, and per-layer attention uniformity, none of which appear on a parameter-count summary.

NOT claimed. This is not a benchmark report and not a ranking of inference frameworks. It does not claim the surviving MoE candidate is "the fastest model" in any absolute sense — only that among the candidates exercised, it is the one satisfying all five constraints simultaneously. The five rules are necessary conditions observed empirically, NOT a proof of sufficiency; a model satisfying all five could still fail on a sixth, unexercised axis. No claim is made about prefill, long-context, batched/aggregate throughput, other Apple Silicon variants, or non-4-bit precision. The two projected ceilings are NOT measured results.

2. Methods

3. Results

Measured (live, byte-parity-gated) single-stream 4-bit decode ceilings and the wall each candidate hit:

Supporting sub-measurements:

• Sparsity isolation (Rule 1): dense ~32B (2.9 tok/s) versus the ~30B MoE on the identical path — a ~2.37x sparsity gain (dense-active vs ~3B-active, ~6.83 tok/s on the comparison path).
• Format wall (Rule 2): the ~35B candidate in its non-aligned 4-bit block format hit a vector-load wall at ~2% of bandwidth; forcing a wide-vector load gave ~1.74x (~24 GB/s) but broke parity to cosine 0.357 (misaligned device load is undefined behavior on this GPU), forcing a scalar fallback to preserve parity. The two aligned 4-bit formats both reach a ~95% load ceiling — no wall — purely on byte alignment.
• Small-matrix wall (Rule 3): the fine-grained-expert family's effective bandwidth stuck at ~7% of peak (~51→56 GB/s); a cooperative-GEMV lever moved tok/s only ~+7% (13.8→14.6, best pairing ~16.5); the GPU was busy (~0.92-0.97 occupancy) but inefficient — a structural ceiling, not a defect.
• ~35B root cause: the ~8x gap to the 30B's 38.8 is NOT dispatch occupancy (already saturated at ~2% of 819 GB/s after grouped-expert GEMV) and NOT continuous-batching overhead (unchanged, GPU ~86% of wall) — it is the 4-bit weight block format being non-4-byte-aligned. This was confirmed across three honest narrowing rounds (a "CB-bound" hypothesis refuted → an "occupancy-debt" hypothesis → root cause = 4-bit format misalignment).

4. Ablations / negative controls

• Format negative control (Rule 2, decisive): forcing the fast wide-vector load on the misaligned 4-bit block collapsed parity to cosine 0.357 — directly demonstrating the misalignment is load-bearing and that the 4.74 ceiling is a format property, not a kernel-maturity deficit. The honest correction chain (CB-bound → occupancy-debt → format) is itself an ablation: each prior hypothesized cause was instrumented and refuted before the format root was isolated.
• Isomorphism falsification (meta-rule): the ~26B MoE is isomorphic to the surviving ~30B MoE on the spec sheet (same top-k routing, similar expert count, similar intermediate size, ~3B active). Ground-truth tensor layout falsified the isomorphism four ways at once — stacked-3D experts vs per-expert-2D, an always-on per-layer dense MLP alongside routed experts, a different gated-activation variant, and a 5:1 alternation of two different attention geometries — yielding the opposite runtime fate. This is the empirical basis for "architecture ≠ runtime compatibility."
• Sparsity-necessary-but-insufficient control: the ~35B hybrid, the fine-grained-expert family, and the ~26B MoE are all ~3B active (same sparsity class as the survivor) yet each lost to a different downstream wall — isolating low-active as necessary but far from sufficient.

5. Threats to validity / limitations

• Single run, single machine, single precision. Every measured ceiling is one run on one M1 Ultra at 4-bit single-stream decode. Per-condition variance, repeated trials, and independent reproduction were not recorded. Hence the evidence level is experimental, not reproduced. Other Apple Silicon variants are out of scope.
• Two ceilings are projections, not measurements. The ~31B dense (~9-12) and ~26B MoE (~18-26) are projected from measured sub-components and labeled as such; they must not be read as live tok/s. The landed numbers (2.9, 4.74, 6.83, 14.6, 38.8) are the only measured ones.
• Necessary, not sufficient. The five rules are empirically observed necessary conditions. Satisfying all five is no guarantee of speed; a sixth, unexercised axis could dominate. The map grows only by the same method (port, measure, find the wall).
• Decode-only scope. Prefill, long-context, and batched/aggregate throughput have different binding constraints. Notably, long-context rewards the SSM-state O(1) memory that made the ~35B hybrid a memory winner (~4.7x @128K) even as a throughput loser — so a throughput-loser under this spec can be a winner on a different axis.
• Survivor framing. The "surviving candidate as accidental exemplar" is a statement about the candidate set exercised, not a proof that no other open model fits the feasible region. The exemplar was not designed for Apple Silicon; it happens to land in the region.
• Rule 3 caveat is load-bearing. Small experts are NOT inherently fatal — the survivor has small experts yet reaches 38.8 because it pairs them with grouped-expert GEMM and an all-layer single-encoder. The rule is therefore conditional ("small experts only viable inside a grouped + single-encoder runtime"), not a blanket prohibition.

6. Reproducibility

• Environment: Apple M1 Ultra, 48-core GPU, ~819 GB/s, 128 GB; custom Metal decode engine, no vendor BLAS. A memory-safety constraint binds: only one 30B-class bf16 model resident (~60 GB) at a time; bf16→4-bit resave is serialized.
• How to reproduce a wall: port the candidate into the custom Metal engine, quantize to 4-bit, run single-stream decode under the byte-identical greedy parity gate, profile GPU timestamps, and read off effective bandwidth / tok/s at the binding stage.
• Caveat: exact command invocations, seeds, and per-run logs are not recorded in this writeup; the reproduction recipe here is method-level rather than turnkey.