The direct HF-logits parity harness (and the compressed-tensors version trap)
Ran Tao (Octoryn Research)
摘要
A correctness-validation method for inference engines: the engine dumps per-token logits, and an offline harness compares them against a HuggingFace transformers oracle on identical token ids with the same weights, gated on cosine similarity with argmax agreement reported. Two small dense families pass cleanly (cosine well above 0.9999). A process finding: a 30B AWQ compressed-tensors per-expert int4 model yields a garbage oracle unless the loader is version-pinned, because newer stacks silently random-initialize the experts. Parity is not quality; the sample is a small dense smoke test.
本 Research Object 以其原始语言(英文)发表。
1. Contribution
Claimed. This note documents a correctness-validation methodology, not a novel algorithm. An inference engine emits per-token logits via a dump mode, and an offline harness compares them row-by-row against a HuggingFace transformers oracle run on the identical token ids with the same weights, gating on cosine similarity and reporting argmax agreement. Because both sides see the same ids and weights, a high cosine is evidence that the engine's forward arithmetic matches the reference math. Two dense families pass cleanly: Qwen2.5-0.5B and SmolLM2-360M, both run against an engine linked only to the GPU runtime and system libraries (no vendor BLAS). A second contribution is a process finding: a 30B AWQ model stored as compressed-tensors per-expert int4 silently produces a garbage oracle unless the loader versions are pinned.
NOT claimed. This is a numerical parity smoke test, not a benchmark-scale evaluation. It does not claim generation quality, downstream task performance, throughput, or that the engine is correct on prompts or positions outside the tested set. Cosine parity on logits says the forward math agrees with the oracle on these inputs; it says nothing about sampling behavior, long-context correctness, or quality. The altitude here is engineering correctness-validation, not a research algorithm.
2. Methods
| Field | Value |
|---|---|
| Hardware | Discrete AMD GPU plus Apple Silicon |
| Stack | Sovereign engine linked only to the GPU runtime and system libraries (no vendor BLAS / DNN libraries); HuggingFace transformers oracle |
| Oracle / baseline | HuggingFace transformers (recent version) for dense families; a specifically pinned older transformers + compressed-tensors pair for the 30B-AWQ case |
| Procedure | Engine dumps per-token logits; an offline harness runs the same token ids through the HF oracle and compares row-by-row |
| Precision | Not recorded in source notes |
| Metric + gate | Cosine similarity per token row (exact numeric gate not stated in source notes); argmax agreement reported |
| Stimulus / inputs | Identical token ids fed to both engine and oracle; same weights on both sides |
3. Results
| Family | Cosine | Argmax | Notes |
|---|---|---|---|
| Qwen2.5-0.5B | >= 0.999979 | not recorded in source notes | engine linked only to GPU runtime + system libs (no vendor BLAS) |
| SmolLM2-360M | >= 0.999998 | 12/12 | engine linked only to GPU runtime + system libs (no vendor BLAS) |
4. Limitations / Threats to validity
- Sample size. Two dense families, each at a handful of token positions (SmolLM2 reports 12/12), is a correctness smoke test, not a benchmark-scale evaluation. No statistical power; no held-out prompt suite.
- Single-run, no variance. Cosine values are point measurements; no repeated runs or variance reported.
- Parity != quality. A passing cosine confirms forward-math agreement with the oracle on these ids only; it does not establish generation quality, sampling correctness, or task performance.
- Oracle trust is load-bearing. The 30B-AWQ case shows a parity number is only as valid as the oracle: a 30B AWQ compressed-tensors per-expert int4 model loaded under a newer loader stack silently random-initializes the experts, producing a garbage oracle and a meaningless comparison. The oracle MUST be validated for real weight loading before any parity number is trusted.
- Scope / quantization bounds. Demonstrated only on small dense models (~0.36B-0.5B). The 30B-AWQ MoE case is documented as a loader caveat, not as a passing parity result in this note.
- Unrecorded fields. Hardware detail, precision, and the exact numeric cosine gate are not recorded in the source notes; an evidence level of reproduced should be read against this thin recorded provenance.
5. Reproducibility / Provenance
- Engine dump: run the engine in per-token logit-dump mode on the target prompt to emit per-token logits.
- Oracle: run the same token ids through HF
transformers. For dense families, a recent stack suffices. For 30B AWQ compressed-tensors per-expert int4, pin a matched oldertransformers+compressed-tensorspair — newer versions random-init experts. After pinning, confirm the oracle loads real weights and emits coherent output before trusting any parity number. - Sovereignty: the engine links only against the GPU runtime and system libraries (no vendor BLAS / DNN libraries).
- Provenance note: exact build and run commands are not recorded in the source notes.
声明边界
作者对范围的明确界定——本工作证明了什么、未证明什么——沿用自 Octoryn Research 的发表模型。
证明
- A per-token logit dump compared offline against a HuggingFace oracle on identical token ids and the same weights, gated on cosine similarity, is a sound forward-arithmetic parity methodology.
- Two small dense model families pass cleanly under this method, with per-token cosine similarity well above 0.9999 and full argmax agreement on the tested positions.
- Oracle trust is load-bearing: a version-validated loader is required, because some quantized models silently random-initialize experts under newer loader stacks, yielding a meaningless comparison.
未证明
- That cosine parity on logits implies generation quality, sampling correctness, or downstream task performance.
- Correctness of any oracle that was not itself validated for real weight loading and coherent output.
- Correctness beyond the tested set: two small dense families at a handful of token positions is a smoke test, not a benchmark-scale evaluation.
- Throughput, latency, or behavior at model or quantization scales other than the small dense models shown.
适用于
- Validating a sovereign inference engine's per-token logits against a separately validated HuggingFace oracle run on identical token ids with the same weights.
- Confirming that an oracle actually loads real weights before any parity number derived from it is trusted.
不适用于
- Comparing against an unverified oracle, or treating a passing cosine as a claim about generation quality or task performance.
- Extrapolating from the small dense sample to large, mixture-of-experts, or quantized models, or to benchmark-scale evaluation.
作者
- Ran Tao — 调查研究, 写作
引用本文
引用格式
Tao, R., Octoryn Research. (2026). The direct HF-logits parity harness (and the compressed-tensors version trap) (TR-2026-0018). Octopus Research Institute.
BibTeX
@techreport{oritr20260018,
title = {The direct HF-logits parity harness (and the compressed-tensors version trap)},
author = {Tao, Ran and {Octoryn Research}},
institution = {Octopus Research Institute},
year = {2026},
note = {Permanent ID TR-2026-0018. Not peer reviewed.}
}披露
- 资助
- 硬件与基础设施由 Octoryn / Octopus Core Pty Ltd 提供。
- 利益冲突
- Octoryn 提供商业推理与治理工具;相关发现独立报告。