Per-stream KV-slot isolation for parallel low-bit decode on Apple Silicon: crash fix with byte-identical solo parity and no-regression

STATUS: Experimental, single measured run. All correctness gates green on a real 4-bit 30B-class MoE model — solo byte-identical, ragged six-concurrent no-crash plus byte-identical, single-stream throughput at baseline. Concurrent aggregate throughput was observed only under contention and a clean number is deferred.

1. Contribution

Claimed. A per-stream KV-slot isolation refactor for an Apple-Silicon (desktop-class) 4-bit quantized 30B-class MoE serving path that fixes a concurrent-decode crash and enables true-parallel (ragged join/finish) decode. The batched KV arena is re-keyed by session, max-stream count, and geometry, and indexed by a stable per-stream slot — never wiped on a batch-composition change. The slot lifecycle is seed-once-at-join, a one-time single-stream-to-batched on-device migration copy when a second job arrives (the reverse direction is forbidden), and release-on-retire (logical reset, buffer reused). The change is additive: the legacy single-stream path is byte-for-byte intact (only the KV-slab offset is re-indexed through the slot map, and the per-call wipe is removed). On the real 4-bit model the solo byte-identical gate, the ragged six-concurrent no-crash + per-stream byte-identical + coherent gate, and the single-stream throughput-at-baseline gate all PASS.

Not claimed. No clean concurrent-aggregate throughput number is claimed — the one multi-stream throughput observation was contaminated by GPU contention from a co-running generative-model session and is deferred. Equality of a migrated (single-stream-to-batched) stream to a pure reference is not claimed beyond no-crash, determinism, and coherence. The initial version did not reconcile a pre-existing single-stream-vs-batched floating-point accumulation divergence (closed in the follow-up below). No variance or multi-window reproduction is claimed.

2. Methods

3. Results

All on the real 4-bit 30B-class MoE model, versus the captured mainline baseline:

• Solo byte-identical: PASS. A truly-solo greedy request (several prompts) is byte-identical to baseline and deterministic within each. The solo path stays on the single-stream kernel by construction; this is a regression guard confirming the refactor did not disturb it.
• Ragged join/finish concurrency: PASS. Under six concurrent ragged (continuous join/finish) requests the prior path crashes; the refactor survives. All six streams complete, are byte-identical to each other and to the solo reference, and are deterministic across repeats. The crash is fixed and concurrent decode is correct through the slots.
• Throughput: PARTIAL. Single-stream throughput equals the baseline (the single-stream kernel path is unaffected) — PASS. A small concurrent run completed correctly but its wall-clock was contaminated by GPU contention with an active co-running generative-model session (cut short to relieve memory pressure with two large 4-bit models co-resident). A clean concurrent-aggregate throughput number is deferred to a clean-GPU window. Non-regression also holds by construction: only the KV-slab offset is re-indexed and the per-call wipe is removed, so per-token kernel work is identical.
• Sovereignty: unchanged. Pool bookkeeping and buffer offsets only; no kernel/BLAS change.

4. Ablations / negative controls

• Falsify-first reference capture. References were captured from current mainline before any hot-path edit.
• Crash contrast as control. Prior path versus refactor on the identical six-concurrent reproducer: prior path crashes, refactor survives — isolating the wipe/re-key as the crash cause.
• No additional negative control (e.g. forced legacy-path divergence) is recorded.
• The pre-existing single-stream (batch-of-one) versus batched (batch>1) tiny greedy divergence is an existing fact, not introduced or reconciled in the initial version.

5. Threats to validity / limitations

• No clean concurrent-aggregate throughput. The only multi-stream throughput observation was contended by a co-running generative-model session and cut short for memory pressure. The non-regression argument for it is by-construction, not measured under a clean GPU.
• Large-batch per-stream byte-identical sub-gate not empirically run in this window (only single-stream throughput plus the solo and concurrency correctness gates were measured).
• Single run, no variance on the throughput number; evidence level is experimental, not reproduced.
• Migrated (single-stream-to-batched) streams equal neither pure reference in the initial version — a hybrid stream inherent to the two-kernel design. The initial version promises only no-crash + determinism + coherence for them.
• Memory standing cost of the fixed max-batch arena (a few gigabytes at a small max-stream count and a multi-thousand-token context) must be reported at boot; requests exceeding the decode context cap are rejected at admission (honest bound), not truncated.
• Co-residency hazard. Runs were GPU-staggered against the co-running generative-model session.

6. Reproducibility

• Method. Solo parity: greedy, fixed seed, several prompts, compared against mainline references. Concurrency: a ragged join/finish reproducer with at least six concurrent streams and several repeats for determinism. Baseline references captured from current mainline.
• Config. A max-stream count and a decode context cap size the fixed arena. Sovereignty verified by checking no vendor BLAS is linked and zero BLAS calls occur on the device path.
• Hardware specifics (exact Apple-Silicon SKU, RAM) not recorded beyond "Apple Silicon, desktop-class". The clean-GPU concurrent-aggregate throughput run is not recorded (deferred).

Follow-up update — single-stream≡batched floating-point divergence eliminated

The single-stream-vs-batched greedy floating-point accumulation divergence that the initial version declared out of scope is now closed. Root cause, at a high level: the batched 4-bit decode path and the single-stream path used dense-projection GEMM variants with different intra-lane reduction associativity over a wide contraction; with identical dequantization, the differing accumulation order produced roughly one-ULP-per-element differences that compounded across the model's layers and could flip a near-tie greedy argmax (a documented trailing-whitespace token). Fix: the batched path selects the reduction order matching the single-stream path when geometry allows, so at batch-of-one the batched path reduces in byte-identical order to the single-stream path. The fused all-layers decode kernel is untouched.

Empirical gate (real 4-bit 30B-class MoE, greedy fixed seed, several prompts): with the unify toggle on, a lone request routed through the batched slot path is byte-identical to the single-stream reference across all prompts (token-level max-abs-diff of zero); and several concurrent batched streams are each byte-identical to the same single-stream reference — so a migrated single-stream-to-batched stream's batched steps now match what the single-stream path would have produced (the earlier hybrid-stream caveat is resolved at the arithmetic level). DECISION: land the fix but KEEP two kernels — routing a lone request through the batched path costs a few percent steady-state throughput, so the single-stream kernel stays the solo fast path. The divergence that forced the two-kernel split is gone (unification is now possible, proven via a default-OFF toggle); the split is retained purely as a solo-path performance optimization, not a correctness compromise.