A sovereign DP router with mid-stream byte-identical failover

Contribution

Claimed. We describe a hand-written ("sovereign", no third-party proxy) HTTP reverse-proxy router that fronts a data-parallel (DP) inference fleet and performs mid-stream failover: when a backend drops during generation, the router greedily resumes the same request on another backend and continues byte-identically away from floating-point tie positions, so the user does not observe a seam. Routing decisions and failover events are event-logged for audit. The correctness argument rests on a separately measured cross-chip determinism result (greedy decode, byte-identical rate on the order of ~96.9% across chips), not on a measurement internal to this note.

NOT claimed. This note does not claim byte-identical continuation under sampled (non-greedy) decode or at floating-point tie positions; it does not measure failover latency or overhead under production load; it does not claim load-balancing optimality or zero-cost routing; and it makes no generation-quality, throughput, or benchmark-scale claim. Honest altitude: this is an engineering/architecture report whose central correctness property is inherited from a cross-chip determinism benchmark. The byte-identical rate is the only quantitative result and it was produced by a different study; no new measurement of the router's own behavior (success rate, latency, seam-freeness rate) is recorded.

Design

Evidence / Status

The router's seam-free claim is derived, not directly measured here. The supporting evidence is a cross-chip determinism benchmark referenced by this note:

The logical bridge: because a continuation computed on a different backend matches what the original backend would have produced (greedy decode plus cross-chip determinism), a dropped stream can be picked up without a user-visible seam at non-tie positions. The router itself is presented as a prototype — no router-internal measurement (failover success count, latency, observed seam rate) is recorded.

Limitations

• Inherited (not internal) evidence. The single quantitative anchor (byte-identical rate ~96.9%) comes from a different benchmark. This note records no measurement of the router itself — no failover success rate, no latency/overhead, no count of seams observed in practice. The seam-free property is argued, not demonstrated.
• Determinism is not 100%. At the small fraction of positions that are not cross-chip byte-identical (sub-ULP floating-point ties), a post-failover continuation can diverge from what the original backend would have emitted. The "byte-identical" claim is explicitly scoped to non-tie positions under greedy decode.
• Greedy-only. Sampled (non-greedy) decode is out of scope; the byte-identical continuation argument does not hold there.
• No load/perf characterization. Failover latency, overhead under concurrent production load, and load-balancing optimality are unmeasured.
• Parity != quality. Byte-identical continuation is a determinism/correctness property; it says nothing about generation quality.
• Evidence level. Carried forward as "experimental"; note that the experimental anchor lives in the referenced determinism study, not in this note's own data. A stricter reviewer could classify this note alone as design/architecture with inherited evidence.

Provenance / Relation

• Relation. Builds directly on the cross-chip determinism result (greedy decode byte-identical rate ~96.9% across chips; exactly identical away from floating-point ties). That result is the load-bearing dependency for the failover-correctness argument.
• Build/runtime. Compiled-from-source release build on commodity Apple Silicon (M-series) hardware; the runtime uses only the platform's native linear-algebra primitives, with no third-party ML framework dependency.
• Reproducibility caveat. Validate the cross-chip determinism oracle first — the failover correctness claim is only as strong as the byte-identical determinism measurement it inherits.