A falsify-first root cause for a concurrent 4-bit decode crash: batch-composition KV-pool wipe trips a re-seed precondition

1. Contribution

Claimed. A code-level root cause, established falsify-first, for a fatal trap (a runtime precondition failure) that killed a single-resident 4-bit ~30B sparse mixture-of-experts (MoE) decode daemon when multiple decode streams ran concurrently. We contribute: (a) a minimal, orchestration-free reproducer (pure HTTP requests against the resident daemon, no agent stack) that isolates the trigger to the resident inference backend; (b) a four-cell reproduction matrix that localizes the crash to concurrent decode and refutes an initial "embeddings poison decode state" hypothesis; (c) the precise mechanism — resident key/value (KV) pools on the GPU matrix-multiply layer are keyed by stream identity and batch size and are wiped on any batch-composition change, after which re-seeding demands a full KV prefix that steady-state decode never carried; (d) a contained mitigation (a feature flag that serializes GPU decode) that makes multi-sub-agent orchestration safe.

NOT claimed. No parallel-decode speedup (the mitigation serializes GPU decode). No fix to the underlying pool design (a per-stream isolation rewrite is unimplemented). No behavior beyond a single machine. No reproduction across multiple machines, model families, or precisions. This is a single-machine diagnostic with a single observed reproduction per scenario; it is not a throughput benchmark.

2. Methods

3. Results

Reproduction matrix (real 4-bit ~30B model, six concurrent streams, one observed run per scenario):

Two crash sites, one cause. Both the batched decode path and the single-stream decode path fail on a precondition with the same intent ("key history too short to seed"). One fires in the batched decode loop, the other in the single-stream cached-step path.

Root cause. The resident KV pools are persistent mutable state on the GPU matrix-multiply layer, keyed by stream identity (single-stream) and by batch size (batched slabs). They assume a stable in-flight stream set. The batch loop performs continuous batching (active, non-finished streams only), so the batch size changes as requests join and finish; a batch-size change wipes the slabs, resets the filled counter, and the seed loop then tries to re-seed every continuing stream up to its current position. Because decode-appended K/V lives only in the GPU pools — the host-side cache is merely a counter — steady-state decode carries only the incremental KV, not the full prefix, and the precondition fails. The single-stream precondition fires symmetrically when a stream migrates between the batched and single pool sets as the active count crosses one.

Why the existing lock did not help. The prior execution-serialization lock prevents a GPU data race but not the state collision: a batch-size change or path migration between locked calls still wipes a continuing stream's KV.

Mitigation (shipped, no engine hot-path code). A feature flag disables single-encoder batched decode, which forces request-granular serialization: the batch loop truncates each step to the oldest job until it finishes, so exactly one stream's pool is live at a time and no wipe collision occurs. Observed: the concurrent-chat scenario with the flag → SURVIVED; the orchestration capstone regression at low max concurrency → all checks PASS, daemon SURVIVED, with all governed sub-agents correct (a calculator action, a memory action, a denied send-email action, and a summarize action).

4. Ablations / negative controls

• Refuted hypothesis (recorded honestly). The initial hypothesis "embeddings poison decode state" is falsified by the matrix: concurrent embeddings SURVIVED while concurrent chat completions CRASHED. The trigger is concurrent decode, not embeddings.
• Lock-domain control. The prior execution-serialization fix does not prevent the crash, isolating the failure to shared-pool state composition rather than a GPU data race.
• Lockstep negative case. A prior constant-batch-size batched-serving benchmark never crashed because it ran all streams in lockstep at a fixed batch size, never triggering the batch-size-change re-seed — showing the crash is specifically a batch-composition-change phenomenon, not batching per se.

5. Threats to validity / limitations

• Single observed run per scenario; no variance, no repeat counts recorded. Evidence level is experimental, not reproduced. The matrix verdicts are crash/no-crash observations, not statistics.
• Hardware under-specified. Exact machine SKU, RAM, and OS version are not recorded in source.
• Single model, single precision. Only the one ~30B sparse-MoE 4-bit configuration was exercised; no claim for other families, precisions, or model sizes.
• Mitigation trades throughput for safety. The serialized-decode mode is concurrent submission with serialized GPU decode: no parallel-decode speedup, and micro-batch throughput is off in this mode. It is a per-deployment choice, not a default flip; the default keeps batched throughput for single-stream and lockstep-batch serving but remains unsafe under the agent-concurrency pattern.
• Root cause is unfixed. The real single-machine fix (per-stream KV/scratch isolation) is a change to the validated batched path and is not implemented; it would require re-validating batched throughput and per-stream byte-identical parity. A scale-out alternative (multi-resident data-parallel federation) is cross-machine only, since one machine cannot hold two copies of the model resident.
• Capstone scope. The PASS at low max concurrency demonstrates safety and correctness under the mitigation; it does not establish an upper concurrency bound or sustained-load durability.

6. Reproducibility

• Reproducer. Run the concurrency-decode reproducer against a fresh 4-bit ~30B resident daemon; observe the two concurrent-decode scenarios crash and the sequential and embeddings-only scenarios survive. Confirm the mitigation by enabling the serialized-decode flag and re-running the concurrent-chat scenario (→ SURVIVED) and the orchestration capstone regression at low max concurrency (→ all checks PASS).
• Environment. Machine SKU / RAM / OS / toolchain versions: not recorded in source.