A multi-week debugging retrospective on an Apple Silicon 4-bit inference engine: command-buffer folding, an encoder leak, and cold-bank reservation

1. Contribution

Claimed. This is an engineering-retrospective technical report on a multi-week debugging campaign in an Apple Silicon 4-bit (quantized) inference engine on the Metal backend. It documents a chain of faults and their fixes: a command-buffer folding mechanism that exposed an encoder memory leak; the leak fix (which had manifested as a process abort / signal-6-style crash) that in turn exposed a cold-bank allocation throw; and the cold-bank fix via a pre-expanded memory-bank reservation plus a precondition guard. The campaign landed a production configuration that, on real model shards, reproduces near-exact numerical parity with the reference (cosine essentially indistinguishable from 1.0), a parity PASS verdict, a third-party-ML-framework call count of zero (a self-sufficiency invariant), and a concurrency measurement of single-stream decode versus batch-4 aggregate throughput at roughly 1.40-1.43x.

NOT claimed. This note does NOT claim the higher single-stream throughput result reported in a separate, later benchmark (cited, not measured here). It does NOT claim the earlier-reported higher concurrency figure, which is honestly corrected downward to roughly 1.40-1.43x. It does NOT claim anything about generation quality — parity cosine is a numerical-equivalence metric, not a quality metric. This is a correctness-validation and bug-fix engineering result and a retrospective, not a novel algorithm or a benchmark-scale performance evaluation.

2. Methods

3. Results

Fault-chain timeline

4. Honest correction (negative finding)

Mid-campaign a NaN was hypothesized to originate from reallocation during encoding. On closer inspection that reallocation was found to be benign backend behaviour, not the fault; the NaN root cause lay elsewhere. The discarded hypothesis is recorded deliberately as part of an honest retrospective. This is a corrected hypothesis, not a controlled ablation.

5. Threats to validity / Limitations

• Sample size. Parity is reported as a single cosine value on real shards; the source does not record how many prompts or token positions back it. This is a correctness smoke test, not a benchmark-scale evaluation.
• Single run / no variance. The throughput numbers are single-run figures with no variance, confidence interval, or repeat count recorded.
• Parity != quality. Near-exact parity cosine establishes numerical equivalence to the reference, not generation quality.
• Precision / scope bound. Results hold for the 4-bit quantized path on the Apple Silicon Metal engine only; they do not transfer to bf16, other quantizations, or other backends.
• Baseline weakness. The concurrency ratio is internal (batch-4 versus the engine's own single-stream), not against an external serving system.
• Superseded headline numbers. The campaign's concurrency figure (~1.40-1.43x) and aggregate throughput are NOT the project's final single-stream result; the higher single-stream figure belongs to a later note. The earlier higher concurrency figure is retracted.
• Stack provenance gap. Exact OS / toolchain / backend versions, shard identity, oracle version, and prompts are not recorded in the source notes, limiting independent reproduction.
• Evidence level. The reproduction scope is the production-config parity run described above, not a multi-seed throughput reproduction.

6. Reproducibility / Provenance

• Build invariant. A release build of the engine was used; the self-sufficiency invariant is zero third-party-ML-framework calls (vendor-supplied accelerated primitives only). Exact version pins are not recorded in the source notes.
• Validate the oracle first. The parity reference / oracle version is not recorded in the source notes; any reproduction should pin and validate the oracle before trusting the cosine gate.
• Relation. This campaign set up a later single-encoder throughput improvement; the higher single-stream figure is documented in a separate single-stream decode benchmark note.