A sovereign ARM-NEON CPU inference backend (Raspberry Pi 4B): Q8 and multicore are complementary, not additive

1. Contribution

Claimed. We brought up a CPU-only text inference backend from zero on the Raspberry Pi 4B (Cortex-A72, ARMv8.0-A) as pure hand-written ARM-NEON host C++ with zero BLAS/HIP/CUDA (a dynamic-library check shows only the C/C++ runtime and math library). Two dense decoder architectures run end to end without external math libraries: a ~0.36B Llama-style model (no attention bias) and a ~0.5B Qwen2-style model (with QKV bias). The load-bearing research result is a composition finding on a bandwidth-starved edge target: narrow weights (fp16/Q8) and multicore threading are complementary, not additive. Neither quantization alone nor threading alone moved decode throughput; only Q8 combined with 4 threads won (~3.65x). This echoes an earlier GPU-side observation that Q8 is "memory-mode-positive / compute-mode-negative," now reproduced on a CPU-only backend.

NOT claimed. We do NOT claim a production-serving stack, batched/chunked prefill, cross-box single-stream tensor/pipeline parallel, MoE/SSM/RWKV/fused-QKV coverage, sustained-thermal endurance, or multi-run statistical variance on the throughput numbers. The throughput figures are single measured runs under one governor/clock condition. The two-box figure is aggregate throughput over independent requests, not a single-stream speedup.

2. Methods

3. Results

Correctness / parity (vs a recent transformers fp32 oracle).

Kernel micro-parity (NEON vs scalar/double reference, on the dev Mac and both A72 nodes): GEMV f32/f16/q8 max_abs ≤ 1e-5; norm/rope/swiglu/gelu max_abs < 3.4e-7; attention decode (GQA) max_abs < 1e-7; composed 2-layer block cosine = 1.0. Generations are byte-identical across both Pi nodes on a fixed prompt.

Throughput — the composition finding (Llama-style ~0.36B decode, Node A 4 cores @ 1.8GHz, greedy, single run).

Mechanism:

1. fp32 is memory-bandwidth-bound. Roughly 1.45 GB of weight traffic per token saturates the single LPDDR4 bus; adding cores does not help (they stall on the same bus) and CPU clock does not matter.
2. Q8 single-thread is a wash (~2.69 vs fp32 2.84). A72 has no int8 dot-product, so the int8→fp32 dequant adds roughly 2x the vector ops of fp32's fused-multiply-add, approximately cancelling the 4x weight-bandwidth saving. Q8 alone is a memory-footprint win, not a speed win.
3. Q8 + 4 threads is the win (~3.65x, 9.83 tok/s). Cutting weight traffic to ~362 MB/token frees enough bandwidth that the workload flips compute-bound and the 4 cores parallelise. The speedup tracks bytes-freed monotonically (none → 1.7x → 3.65x).

2-node data-parallel (Llama-style ~0.36B q8 / 4T). Concurrent independent streams: Node A 9.75 + Node B 8.33 = ~18 tok/s aggregate. This is aggregate throughput over INDEPENDENT requests (data-parallel), NOT one stream made faster.

4. Ablations / negative controls

• Clock-sensitivity flip (the mechanism's confirming control). Under Q8+4T the workload is clock-dependent: Node B @ 1.5GHz = 8.40 tok/s, ~14% below Node A @ 1.8GHz's 9.83, matching the 1.5/1.8 clock ratio. Under bandwidth-bound fp32, clock does not matter. This flip is the positive evidence that Q8 moved the workload from bandwidth-bound to compute-bound.
• rms_norm_eps perturbation: moving the norm epsilon far off its configured value collapses cosine to −0.67 and the output degenerates to "is is is is". Proves the norm path is load-bearing and config is honoured. (Source note: the Llama-style config ships eps=1e-5, NOT 1e-6 — a recurring trap.)
• rope_theta perturbation (1e5 → 1e4): argmax flips (Paris → the), text degrades. Honest caveat: cosine only dips to 0.9999 at 5-token context (RoPE angles are tiny there), so the argmax flip — not cosine — is the discriminating signal; cosine is weak on short prompts.

5. Threats to validity / limitations

• Single-run throughput, no variance. All tok/s figures are single measured runs under one governor (performance) / one clock condition. No multi-run mean/std is recorded — hence evidence level is experimental, not reproduced. (Correctness, by contrast, was confirmed byte-identical across two boxes.)
• Decode only (M=1). Prefill reuses the same per-token path; no chunked/batched prefill.
• Coverage. Only two dense architectures (Llama no-bias, Qwen2 QKV-bias). Fused-QKV, scalar-multiplier, MoE, SSM/RWKV not yet wired.
• Kernel correctness-first overheads. Attention recopies a compact KV slice per layer per token (a strided-view kernel would remove the copy); transcendentals use exact libm scalar (vectorized polynomial approximations are future work). These bound absolute tok/s but do not affect the relative composition finding.
• Thermal. Both boxes idle ~45-50°C, no throttling observed; sustained multi-minute decode under load was NOT endurance-tested.
• Bandwidth figure is effective, not bus-spec. The ~4 GB/s effective bus figure underlying the mechanism narrative is an observed-saturation interpretation, not an independently profiled bus benchmark.
• 2-box aggregate is not single-stream. Cross-box single-stream tensor/pipeline parallel is LAN-latency-bound and out of scope.

6. Reproducibility (high-level)

The implementation is a config-driven driver over hand-written NEON kernels, reusing an external-math-library-free safetensors loader and a BPE tokenizer (Llama- and Qwen-compatible), with a header-only JSON dependency vendored to the on-box build directory. Provisioning sets swap and pins the CPU governor to performance; deployment builds on-box with the recent GCC toolchain noted above. Parity is validated by dumping last-token logits and comparing to the transformers fp32 oracle.