Cross-chip determinism: 96.875% byte-identical decode across two chips, live

Contribution

Claimed. Under greedy decode of one identical 4-bit quantized model running on two different accelerator chips of the same architecture, 96.875% of decoded tokens are byte-identical between the two chips, measured live (not a replay). The residual (~3%) is attributable to sub-f16-ULP tie positions — token slots where the top-1 and top-2 logits are equal or within one f16 unit-in-the-last-place, so independent accumulation order on different hardware can resolve the tie to the other index. This is a correctness/consistency validation result that quantifies the floor of cross-chip greedy continuation; it is the empirical basis for mid-stream byte-identical failover in a data-parallel router.

NOT claimed. This is not a novel algorithm — it is an engineering consistency measurement. It does not claim 100% cross-chip determinism (irreducible f16 ties remain), byte-identical behaviour under sampled (non-greedy) decode, determinism across more than the two tested chips, determinism for non-4-bit paths, any statement about generation quality, or any throughput/latency result. Byte-identical token agreement is a determinism property, not evidence of output quality. The sample is a live smoke-scale consistency check, not a benchmark-scale evaluation.

Methods

Results

Threats to validity / Limitations

• Sample size. The source records only that "the same prompts" were decoded on two chips; the prompt count and total token count are not recorded. 96.875% is a single live consistency smoke test, not a benchmark-scale evaluation. The percentage is reported to high precision, but its denominator is not stated.
• Single run / no variance. One live run is reported. No repeated trials, confidence interval, or variance is recorded.
• Two chips only. Determinism is shown for exactly the two tested chips. It does not generalize to other chips or to a third.
• Precision ceiling. The engine path is f16; the residual ~3% is an irreducible f16-tie phenomenon. The "sub-f16-ULP tie" attribution is an interpretation consistent with prior single-engine parity work, not an independently re-measured logit-gap distribution in this note.
• Greedy and 4-bit only. No claim holds under sampled decode, mixed precisions, or non-4-bit paths.
• Parity != quality. Byte-identical token agreement says nothing about whether the generated text is good.
• Evidence level. This rests on the cross-chip live run plus prior single-engine f16-tie documentation. Readers should treat the residual attribution as inherited from that prior work rather than re-derived here.

Reproducibility / Provenance

• Commit(s): not recorded in source notes.
• Build / run commands: not recorded in source notes.
• Version pins: not recorded in source notes.
• Procedure to reproduce (high level): greedily decode identical prompts of the same 4-bit quantized model on two different same-architecture chips and compare per-position token IDs live.
• Caveat: the residual attribution depends on the single-engine f16-tie analysis; validate that prior parity result before relying on the "not drift" interpretation.
• Relation: underpins a data-parallel router's mid-stream byte-identical failover — a dropped chip's request can be greedily continued on another same-architecture chip and, away from tie positions, produce the identical continuation.

Measured cross-backend agreement (this work)

This section reports a NEW, directly-measured number that is distinct from the headline 96.875%. Honest framing up front: the original 96.875% was a same-engine, two-same-architecture-chip result. Reproducing that setup needs a second same-architecture chip we do not currently have. What we can measure is the cross-backend alternative: two independent implementations of the SAME base model decoding the same prompt. That is a strictly harder consistency test (different kernels, different hardware vendor) and yields a different, lower number — reported here without overstating it as the same quantity.

Setup (high level).

• Backend A: a same-architecture on-chip-dequant 4-bit decode path, using its own quantized matrix-vector kernels with no vendor BLAS dependency.
• Backend B: an independent streaming 4-bit decode path on a different hardware vendor, also avoiding vendor BLAS.
• Same base model AND same quant: both load the identical group-quantized 4-bit checkpoint of the same publicly known MoE base model (configuration and shard sizes match). So divergence is purely cross-backend numerics (vendor-specific dequant/matmul/attention/positional-embedding rounding), NOT a quant-scheme mismatch.
• Prompt token stream aligned byte-identically: a short factual prompt is tokenized once and the same integer ids fed to both backends, with chat templates bypassed on both. Greedy (temperature 0 = argmax), 32 new tokens each.

Result — positional agreement rate: 7/32 = 21.875%. Both backends agree byte-identically on the first 7 generated tokens, then diverge at position 7 and never re-sync; one continues toward one factual completion and the other toward a different one.

Disagreement analysis (logit gaps directly dumped, not assumed). The low positional rate is an autoregressive cascade from a single argmax flip at the first divergence, not 25 independent disagreements. Teacher-forcing the identical context (prompt + 7 agreed tokens) and dumping full-vocab logits at the divergence step on both backends:

Both backends rank the SAME two tokens as their top contenders — they merely flip the ordering of a near-tie. The decision is a near-tie on backend A (gap ~0.23) but the cross-backend numeric drift (a few tenths of a logit at magnitude ~28) was enough to invert it. Honest caveat on the tie hypothesis: this is NOT a sub-f16-ULP degenerate tie — the f16 ULP at magnitude ~28 is roughly 0.016, and the observed gaps (~0.23 / ~0.80) are an order of magnitude larger. So the flip is real cross-backend numeric divergence of order a few tenths of a logit in a near-tie pair, larger than an irreducible ULP tie. Once greedy decode commits to one branch, every later position conditions on a now-different prefix, so positional agreement collapses even though the two implementations are numerically close. The faithful summary metric here is the divergence point (token 7), not the 21.875% positional rate.

How this differs from 96.875%. 96.875% = SAME engine on two SAME-architecture chips (byte-identical kernels; residual = sub-ULP ties). 21.875% = TWO DIFFERENT independent backends (different hardware vendors, different kernels) on the same model+quant; residual is honest cross-backend numeric divergence that cascades through greedy autoregression. These are NOT the same measurement, and the lower number does not weaken the original claim — it characterizes a strictly different, harder scenario. The original same-architecture two-chip number remains blocked on a second same-architecture chip.

Cross-backend divergence distribution (n=6 prompts)

The single-prompt result above (first divergence at token 7) is upgraded here from n=1 to a distribution over 6 prompts. Same setup: prompts tokenized ONCE and the SAME integer ids fed to both backends (byte-identical, chat templates bypassed, temperature-0 greedy both sides). Honest caveat: backend B ran in a per-layer streaming mode rather than fully resident because its accelerator could not hold the full model resident. Streaming is mathematically identical to resident (same kernels, same bytes, same argmax), so it does not affect the divergence result.

Median first-divergence position: token ~9.5 across the 6 prompts (the code-ish prompt produced no divergence in its 24-token horizon).

Are all divergences near-tie flips? No. 4 of 5 observed divergences are clean top-2 flips — each backend's argmax is exactly the other backend's rank-2, with the decision gap in the range ~0.06 to ~0.46 logits. These are classic 4-bit cross-backend numeric-noise near-ties. The lone exception is the instruction prompt: the chosen token is only the other backend's rank-3 (cross-gap ~2.3), at a genuinely diffuse end-of-output position — a wider-than-top-2 disagreement, reported rather than forced into the near-tie bucket. The code-ish prompt was fully byte-identical across all 24 tokens (deterministic code structure).

What the distribution shows. This upgrades the finding from a single anecdote to a characterized distribution: cross-backend greedy determinism between two independent implementations breaks early and consistently — median first divergence at token ~9.5, every non-code prompt diverging within the first 17 tokens. The mechanism is small but REAL cross-backend numeric drift (a few tenths of a logit, occasionally larger) that inverts a near-tie, NOT sub-ULP degenerate ties: the typical decision gaps (~0.06–0.46 logits at magnitude ~28, where an f16 ULP is roughly 0.016) are several to many times an f16 ULP, and one case is a full ~2.3-logit cross-gap. This is a distinct and strictly harder phenomenon than the same-architecture 96.875% byte-identical result, and the lower cross-backend agreement does not weaken that original same-engine claim.

Instrumentation note: the measurement instrumentation on both backends was additive and default-off (flag-guarded, no behavior change when unset), and the additive instrumentation was reverted after measurement to keep the working tree clean.