Federation continuity primitives: an event-sourced session log with deterministic replay, cross-machine replication, and self-healing recovery

1. Contribution

Claimed. A distributed-inference continuity layer can be built on a single primitive: an append-only, line-atomic, durably flushed, hash-chained session log that event-sources visible token emissions, not KV-cache or memory copy. From this one substrate, four behaviors are demonstrated at mock/single-process and small-real-model level:

1. Deterministic cold replay. A crash-safe (torn-tail-dropping) line-delimited JSON log is replayed across a fresh process to rebuild the live continuation convention; a monotonic emission cursor gives exactly-once visible emission across the seam.
2. Tamper detection. An opt-in hash chain (each record's digest covers the prior chain head plus the canonical record body) makes any on-disk edit detectable; the chain breaks at the edited record. A digest shim (platform-native cryptography on Apple, a pure-language standard hash on Linux) is shown byte-identical across both architectures via known-answer tests.
3. Cross-machine replication. The verbatim record stream is shipped to a replica store; a surviving node pulls it over the LAN and replays it. Demonstrated surviving both disk loss (deleting the local log) and a full process kill of the coordinator, with the replica running on a second physical machine.
4. Self-healing recovery. A hybrid model: a local per-node supervisor (primary) owns crash-to-respawn, plus a coordinator-side fallback manager that fires only on the health monitor's healthy-to-unhealthy transition, runs only operator-allowlisted commands (never network-derived), and is gated by a per-node circuit breaker with exponential backoff.

Continuation rebuild refuses mismatch. Resume is rejected on model, sampler, or backend-family mismatch — no silent cross-model continuation.

NOT claimed. No real-fleet-under-load measurement. No KV-cache or memory replication (deliberately not built — see §4). No consensus, quorum, or multi-replica arbitration. No production end-to-end at large-model scale across a heterogeneous fleet. The large-model coordinator paths use small real models or mock backends; the cross-machine demonstrations use a small open dense model, never a large model.

2. Methods

3. Results

All figures below are single demonstrated runs unless otherwise stated; no variance is recorded. Internal commit identifiers and configuration names from the source have been omitted.

Replay / continuity. A live multi-gate demonstration with no model: two loopback HTTP/SSE mock backends plus a real proxy engine plus a durable, flushed sink plus a real mid-generation kill plus a real failover plus two separate child processes. All gates pass: the client saw one continuous completion (monotonic cursor, no duplicate); restart-recovery rebuilt the prefix in a fresh process; model mismatch was refused. Unit and router test suites pass with no regression.

Production hardening — durability / rotation / retention / audit. Five gates, each with a unit and live proof: (1) rotation — many small tokens produce more than one segment with contiguous sequence numbers, replayed monotonically and exactly-once across the seam; (2) retention — prunes closed segments only, leaving an unfinished session byte-identical and recoverable; (3) crash-recovery — a real hard kill mid-life yields a clean restart with all durable tokens, at most one torn tail, no corrupt record, chain intact; (4) audit — the hash chain detects an on-disk edit and is queryable by session, node, and time; (5) no-regression — all session-log and router tests pass, and the default-off store is byte-identical to the legacy format.

Cross-node log-shipping replication, disk-loss survival. A multi-gate real-model demonstration (small open dense model) in async and sync modes: all pass, client byte-identical across the full stream — byte-faithful ship; disk-loss survival (delete local log, recover from replica, finish on the second node byte-exact); no duplicate visible token; no silent cross-model; honest consistency; replication-off byte-identical to the prior path. The initial governance rating placed cross-physical-machine replication as not yet shown (replica in-process on the same machine).

Cross-machine replica + physical node-loss. The replica runs on a different physical machine; survival is proven by a full process kill of the coordinator. Multiple gates pass (live small open dense model): cross-machine byte-faithful ship; physical node-loss survival (the second machine drives the surviving node); no duplicate token across the seam; no silent cross-model; real-model byte-exact (zero resume divergence, client equals reference across the full stream); replication-off byte-identical. The digest known-answer test is byte-identical across the Apple and Linux machines. No regression across the test suites.

Self-healing recovery. A live before/after re-test, all pass: local supervisor respawn on the order of a few seconds, zero-manual; the coordinator hook attempts once and succeeds once; the circuit breaker caps at three attempts, gives up, stays quarantined, and does not thrash; a false-positive slow node is never restarted (same process id); a single-instance guard refuses a second resident of the large model. Build clean; router tests pass (existing plus new self-healing tests); the recovery manager is absent for a hook-less fleet, giving a byte-identical health path.

4. Ablations / negative controls

• Rollback (default-off) as negative control. Every feature (chain, rotation, retention, replication, recovery hook, session-log sink) is default-off, and the on-disk and wire path is asserted byte-identical to the prior path when disabled. This is the standing "does adding the feature change the baseline?" control: no.
• Mismatch refusal. A reused-session rerun correctly failed (the replica refuses to clobber); cross-model resume is refused. These are intentional failures, treated as passing gates.
• Slow != dead. A false-positive slow node is never restarted (verified same process id post-test); recovery fires only on a configured number of consecutive probe failures, on a different socket than streaming.
• KV replication — decisive negative result. Cross-chip KV-cache replication was investigated analytically and judged NOT bit-portable: post-positional-encoding floating-point state, non-bit-exact per-chip kernels, and layout divergence. The continuity path is therefore the session-log token prefix (orders of magnitude smaller and exact), re-prefilling native KV on the resume node. KV replication is explicitly not claimed and not built.

5. Threats to validity / limitations

• Mock / small-model scale. The replay demonstration uses mock backends and no model; the cross-machine and replication demonstrations use a small open dense model. There is no real-fleet, full-large-model, concurrent-load measurement. Behavior under production load is unproven.
• Single run, no variance. The recovery latency, gate counts, and byte-equality results are single demonstrated runs. No repeated trials, distributions, or variance are recorded.
• At-least-once, not exactly-once at the durability edge. A single in-flight token may be greedily re-emitted on resume; the exactly-once property is over visible emission via the suppressor cursor. KV is never replicated.
• Single replica is the honest ceiling. State replication is operational with a single replica; there is no quorum, consensus, or arbitration.
• Chain proves integrity, not authorship. The hash chain is tamper-evident (detects edits) and single-writer; it does not prove who authored a record, and is not a multi-writer or Byzantine-resistant ledger.
• Coarse SSE seam caveat. The serving layer streams coarse SSE chunks, so in the cross-machine demonstration the shipped seam was a single record; finer-grained seam behavior under real chunking is not exercised here.
• Local supervisor is a test harness, not a production init system. The local supervisor script is labeled non-production; the production-standard mechanism (a service-manager-shaped supervisor) is the intended target, not what was measured.
• Readiness context. Per the federation validation review, overall federation readiness was conservative at the time of these changes, with state replication operational at a single replica; this report covers the continuity primitive, not the whole federation.

6. Reproducibility

• Build. A clean release build. Tests cover the session-log, router, replication, self-healing-recovery, and federation-state-spike suites.
• Demos. A multi-gate no-model session-log demo; a multi-gate distributed demo on a small open dense model; a cross-machine replication demo in node and coordinator modes with the replica on a second physical machine.
• Crypto / sovereignty. A digest known-answer test (empty / short / multi-block inputs) byte-identical across Apple and Linux; dynamic-dependency inspection of the Linux replica server (standard-library/system only).
• Not recorded in source. Exact host hardware and OS versions of the demonstration runs, wall-clock end-to-end timings beyond the local respawn figure, and any multi-run statistics are not recorded in the cited sources.