Batched-concurrent 4-bit serving on Apple Silicon: weight-stream amortization with an honest sublinear ceiling

1. Contribution

Claimed. A request-level batched-concurrent decode path for 4-bit (packed-int4) serving of a 30B-parameter mixture-of-experts model on Apple Silicon. B concurrent requests' decode steps run through one fused forward per token: all transformer layers plus the per-stream output projection are folded into a single GPU command buffer (one submit/wait for the whole batch), the dense attention and output projections run as a single batched 4-bit GEMM so each packed-int4 dense weight is read once across the B rows, and the selected experts of all B streams are dispatched as one grouped sparse-expert operation against the device-resident expert bank. This amortizes the per-token weight stream over the batch and raises aggregate throughput above request-level serialization. The path is the Apple-Silicon analogue of previously published CUDA continuous-batching work (cf. TR-2026-0019) and builds on an earlier single-stream optimization campaign (cf. TR-2026-0020).

NOT claimed. Not linear scaling in batch size — the central honest finding is that aggregate throughput is sublinear in B and plateaus. Not a claim that batching beats the best single-stream after the single-stream path itself was further optimized (see Sections 3 and 5). Not a multi-run / variance-bounded benchmark — every number here is a single measured run. Not per-stream KV isolation (a separate, later effort — do not conflate). Not a saturation of the GPU's compute or memory bandwidth (decode stays weight/dispatch-bound at low bandwidth utilization).

2. Methods

3. Results

All numbers are single measured runs on the real 4-bit shard, sole resident (~15.4 GB), greedy, no vendor BLAS.

Per-stream correctness (load-bearing). Each stream's output is byte-identical whether decoded alone or inside a batch — the batched GEMM row math is row-independent and per-stream glue runs at per-stream offsets. Reported per-stream greedy token ids match the single-stream oracle for B=1/2/4, and per-stream cosine versus the single-stream oracle is approximately 0.999999999999 (both on the resident model and on a synthetic 4-stream ragged-position test).

Initial wiring. Single-stream throughput is not regressed by enabling the path; B=2 aggregate is about 1.22x; B=4 aggregate reaches roughly 1.39-1.41x over the request-level serialized baseline. Per-token GPU syncs collapse from B to 1; the dense weight read is amortized about 4x over B=4; the number of grouped-expert dispatches drops substantially.

Default-on verification. A subsequent verification confirmed no single-stream regression, B=2 aggregate near 29 tok/s, and B=4 aggregate around 33.5 tok/s versus a serialized baseline near 23.8 (~1.41x). A synthetic assertion confirmed one command buffer per step with the path enabled versus per-stream/per-layer serialization when disabled.

Early concurrency anchor. An earlier-campaign measurement recorded B=4 aggregate at roughly 1.40-1.43x over the single-stream anchor, with per-stream greedy parity passing for B=1/2/4.

Plateau. The batched path extends to about B=8 at roughly 35-37 tok/s (~1.40-1.55x over serialized). Experts are bandwidth-bound and routing is largely disjoint across streams, which is the structural ceiling.

Saturation / honest reversal. Profiled on the real 4-bit 30B model, the batched aggregate saturates at roughly 35.6 tok/s and does NOT beat a separately optimized serialized single-stream of about 37.1 tok/s. Decode-time bandwidth utilization stayed around 11-15%. A parity-safe setup-cache optimization was later ported to the batched path (about +37% at B=1, per-stream byte-identical), but the batched path was kept default-off in that later state. Importantly, this saturation measurement was taken against a more-optimized single-stream baseline than the earlier default-on measurements; they are not the same baseline (see Section 5).

4. Ablations / negative controls

• Off-switch as in-binary oracle. Forcing request-level serialization (per-stream/per-layer batched decode) requires per-stream greedy ids to match the batched path. This parity control passes (B=1/2/4).
• Command-buffer count. A synthetic test asserts one command buffer per token with the path enabled versus the serialized count — direct evidence that the fusion, not the math, changed.
• Expert-dedup negative result. De-duplicating experts across streams regressed to about 0.94x because real routing is roughly 79-97% disjoint, so compaction/restructure cost exceeds benefit; the ~4x amortization holds only for an artificial microbenchmark. Kept default-off.
• Root-cause attribution. Aggregate is sublinear because the 4-bit dense and expert ops are B serial GEMVs, not M-row tiled GEMMs — the per-row cost is dispatch arithmetic, not weight traffic — so adding rows does not amortize past the plateau.

5. Threats to validity / limitations

• Single run, no variance. Every tok/s figure is one measurement under one condition. No multi-run, no confidence interval, no independent reproduction. Evidence level is experimental, not reproduced.
• Shifting single-stream baseline is the biggest caveat. The earlier measurements report B=4 ~ 32.8-33.5 beating a ~23.6-23.8 serialized baseline (1.39-1.41x). A later measurement reports the aggregate plateau (~35.6) does NOT beat a separately optimized serialized single-stream (~37.1). The "win" is relative to the serialized baseline available at the time; once single-stream itself improved, the batched aggregate no longer led. Readers must not combine the 1.41x figure with the 37.1 single-stream figure — they are from different baselines.
• Sublinear ceiling is structural, not a tuning gap. Plateau ~35-37 tok/s (B=8) at ~11-15% bandwidth utilization. The unbuilt lever is an M-row tiled grouped-expert GEMM (a higher-risk kernel rewrite); it was not built or measured here.
• Workload-shape dependence. Gains depend on streams routing to overlapping dense weights (always) but largely-disjoint experts (limits expert amortization). Different routing distributions would shift the plateau.
• Large-batch cliff. Aggregate does not grow monotonically; a separate profiling run recorded a sharp drop at very large batch size. The operating point should stay in the mid-range plateau.
• Scope. A single high-end Apple Silicon desktop class and a 30B MoE 4-bit model only. Not validated on other Apple chips, other models, bf16, or non-greedy sampling. Request-level batched serving only; per-stream KV isolation is out of scope.
• Parity is cosine + greedy-id, not bit-exact logits. Per-stream cosine ~ 0.999999999999 and greedy ids match; this is parity to the in-binary serialized oracle, not an independent full-logit bit-match.

6. Reproducibility (high-level)

• Environment. A high-end Apple Silicon desktop workstation with 128 GB unified memory; the 30B model in 4-bit packed-int4 (~15.4 GB) as sole resident; release build; no vendor BLAS.
• Toggle. A single feature flag selects the batched-concurrent path; disabling it restores the serialized single-stream oracle.
• Synthetic parity (no resident model). A filtered release test asserts per-stream cosine versus the single-stream oracle, the per-token command-buffer count, and the no-vendor-BLAS condition.
• Operating discipline. Only one 30B-resident test at a time on the 128 GB machine; use the 4-bit weights (~15.4 GB), never bf16 (~60 GB); gate on free memory before launching.
• Not recorded. Exact prompt set, generation length per run, warmup-inclusion policy per individual figure, and number of repetitions (single run assumed).