OLMoE: the first sparse-MoE in a dense sovereign engine (and an eps trap)

1. Contribution

Claimed. A hand-written, sovereign sparse mixture-of-experts FFN — a router selecting top-8 of 64 experts, each a SwiGLU, combined as a weighted sum with router weights used un-renormalized (no top-k renormalization) — reproduces OLMoE-1B-7B HuggingFace f16 prefill logits at cosine 0.999999, argmax 8/8 on an AMD RDNA2-class GPU (gfx1030). The result also isolates a load-bearing configuration trap: the model config omits the RMSNorm epsilon; the engine had assumed a value 10x smaller than the HuggingFace default. Because the embedding variance here (~9.5e-6) is comparable to epsilon itself, the 10x error inflated the RMSNorm output by ~1.36x and capped parity at cosine 0.99. Setting epsilon to the correct (larger) default restored cosine 0.999999. This is the first sparse-MoE family in an engine that previously had only dense FFNs.

NOT claimed. This is a correctness-validation (logit-parity) result, not a novel algorithm and not a performance result. It does NOT claim generation quality, MoE dispatch throughput, or any benchmark-scale evaluation. It does NOT claim parity for larger MoE models, many-expert routing at scale, or other MoE families. It does NOT claim that an 8-bit-quantized path matches f16 fidelity (the quantized path measured cosine 0.999341, separately). The epsilon finding is an engineering diagnosis specific to models whose normalized signal variance is near epsilon — not a general claim about all RMSNorm models.

2. Methods

3. Results

4. Ablations / Negative controls

The epsilon comparison functions as a controlled ablation of the normalization epsilon:

This pairing demonstrates the epsilon is load-bearing here specifically because embedding variance (~9.5e-6) is comparable to epsilon.

5. Measured eps-sensitivity curve

To turn the "epsilon is load-bearing because embedding variance ~ epsilon" claim from an anecdote into a curve, a runtime override of the RMSNorm epsilon was added (mirroring an existing negative-control hook used elsewhere in the engine) and the epsilon was swept. Each run dumps the prefill last-position logits; cosine is taken against the HF-correct epsilon reference (itself validated at cosine 0.999999 vs the HF oracle). The relative-epsilon column is normalized so the HF-correct value is 1x. f16, OLMoE-1B-7B, single short prompt, RDNA2-class GPU, single run.

Findings. (1) A sharp peak at the correct epsilon. (2) The engine's original assumed value (0.1x of correct) gives cosine ~0.992, independently reproducing the original "~0.99" degradation and quantifying the cost of the 10x epsilon error. (3) The model tolerates a band of epsilon values for argmax stability (top-1 token preserved) but argmax breaks once epsilon is roughly an order of magnitude above the correct value (top-1 flips, cosine collapses to ~0.85-0.87). (4) The sensitivity is concentrated exactly in the regime where epsilon is comparable to the embedding variance, which is the mechanism: when the normalized signal's variance is near epsilon, epsilon stops being negligible. This is a measured sensitivity curve, not a single before/after point.

6. Threats to validity / Limitations

• Sample size is a correctness smoke test, not a benchmark. Parity rests on a small number of short prompts (exact count not recorded in source notes) over a handful of token positions, reported as argmax 8/8. This is a correctness signal, not a benchmark-scale evaluation of accuracy or capability.
• Parity is not quality. Cosine 0.999999 / argmax 8/8 validates that the engine reproduces oracle logits; it says nothing about end-to-end generation quality, long-context behavior, or downstream task performance.
• No variance / single configuration. Numbers are single-run point measurements; no repeated trials or variance are reported. VRAM figures are single observations.
• Precision ceiling. f16 is the highest precision path reported; the 8-bit-quantized path (cosine 0.999341) is honestly lower fidelity and is not claimed to match f16.
• Scope of the epsilon finding. The epsilon trap generalizes only to models whose normalized-signal variance is near epsilon; it is not a universal claim about RMSNorm.
• Baseline weakness. The oracle is HuggingFace transformers at an unrecorded recent version; the source does not pin the version or the exact cosine gate, so reproduction requires re-establishing the oracle. Validate the oracle (real weight load, coherent output) before trusting parity.
• Quantization / family bounds. Results apply to OLMoE-1B-7B on gfx1030; no claim for larger MoE, scaled routing, or other MoE families.

7. Reproducibility / Provenance