Qwen3.6-27B vs llama.cpp — parity rollup (2026-06-28)
This page reconciles the scattered Qwen3.6-27B (qwen35 arch, hybrid Gated-DeltaNet)
parity status into one document an auditor can read top-to-bottom. It does not introduce
a single new number: every figure traces to a file cited inline, and every gated row
names the host capability it needs and the exact one command that produces its witness.
Provenance / honesty rule. This rollup was assembled on a windows/amd64
orchestrator host with no Apple Silicon, no NVIDIA GPU, and no 27B artifact. Per
docs/proofs/00-METHOD.md, every speed / GPU / 27B figure
below is a recorded prior witness from a Mac (or AMD/Vulkan) node, never re-measured
here. A SKIP is not a PASS; a gated item is never presented as run-here.
1 — The verdict, in one paragraph (two senses of “parity”)
“Parity” means two different things and they have different states:
- Correctness parity (does fak’s greedy decode reproduce llama.cpp’s token stream?):
PROVEN at the architecture level — the tiny
qwen3_5HF fixture is bit-exact vs HF transformers (cosine 1.000000, max|Δ| ~4e-9, argmax parity at every position) — but REFUTED at 27B scale: on the realQwen3.6-27B.q4_k_m.gguf, fak and llama.cpp match the first two tokens, then fak’s argmax flips on a near-tie at token 3. This holds on both fak paths (GGUF→Q8 and resident-q4k), so it is diagnosed as a kernel-numerics divergence at 27B scale on the hybrid GDN path — not a Q8 round-trip artifact and not a bug in the reference path. - Speed parity (tok/s vs the llama.cpp-Metal bar):
not yet. fak’s own M3 Pro engine runs the 27B end-to-end but its single-stream decode is 0.1 → 0.9 → 1.2 tok/s along its three measured paths, still ~6× under the 7.29 tok/s llama.cpp-Metal bar. The wall is diagnosed (per-call command-buffer launch overhead, not arithmetic) and the kernels are bit-correct; the closing levers are tracked but Apple-Silicon-gated.
The model runs end-to-end in chat through fak’s own in-kernel engine on the M3 Pro (no llama.cpp in the path) — that part is proven; what remains open is correctness at 27B scale and speed.
2 — Proven (host-independent — runs on a plain CPU box, no GPU / no 27B)
These are green on the orchestrator class of host (CPU-only, no weights). They are the durable parity floor.
| What | Witness (test / artifact) | Result | Source |
|---|---|---|---|
| Architecture math bit-exact vs HF | TestOptionalQwen35HybridOracleForwardMatchesHF (tiny qwen3_5 fixture, 3 GDN + 1 gated full-attn layer) |
per-layer hidden-state cosine 1.000000, max|Δ| ~4e-9, argmax parity at every position | FAK-NATIVE-QWEN35-RESULTS.md; QWEN36-PARITY-RESULTS.md §”fak-native status” |
| Tokenizer byte-exact vs llama.cpp | internal/tokenizer oracle gate (#90) |
byte-exact on the Qwen vocab + the 22-token ChatML smoke prompt | QWEN36-PARITY-RESULTS.md |
| GGUF tensor mapping (tiny + real) | TestQwen35GGUFConfigCanonicalizesHybridTensorsAndRunsForward, TestOptionalQwen35GGUFMapsEveryTensorName |
all 851 real-GGUF tensors map; hybrid knobs derived | FAK-NATIVE-QWEN35-RESULTS.md |
| Cached session == cacheless forward | TestQwen35HybridSessionMatchesForwardAndPersistsState, TestQwen35HybridQuantTokenLoopPersistsState |
last-position logits match | FAK-NATIVE-QWEN35-RESULTS.md |
| #71 Metal-hybrid-prefill CPU orchestration | TestQwen35HybridViaMMMatchesCPUTemplate (drives prefillQwen35HybridViaMM) |
logits + KV cache + linear-attn cache match the proven CPU template within ~1e-6 Q8 float-order drift; green on windows/amd64, CGO_ENABLED=0 — re-built + re-run green 2026-06-29 at cf8af435 (#1242 build-hole closure: the prior dogfood’s policy-blocked SKIP is now a real build, not a rounded pass) |
experiments/qwen36/metal-hybrid-prefill-status-2026-06-28.md §2; experiments/agent-live/qwen36-build-hole-closure-20260629T071148Z.json |
| q4_k GEMM/GEMV dispatch bit-identical | TestQ4KGemmMatchesMatRows, TestQ4KGemmInt8MatchesMatRowsInt8, TestQ4KMatRowsMatchesF32 |
batched GEMM bit-identical to per-token decode GEMV (f32 + int8-SDOT) — the q4_k majority adds zero drift | experiments/qwen36/metal-q4k-device-gemm-status-2026-06-28.md §3; …decode-gemv-status… §3 |
#71 model-lane code LANDED on main |
core prefillQwen35HybridViaMM (c80d64fa); Metal twin + gate + stub + kv.go dispatch (5c065118) |
dos commit-audit diff-witnessed (code_effect) |
…metal-hybrid-prefill-status… §1/§3 |
Build note for re-verifying any Go witness here: HEAD often doesn’t build standalone (it
references uncommitted peer fields, e.g. an in-flight forward.go/normWeights
refactor); build the live tree with the peer pkg frozen at clean HEAD (git archive HEAD
into a scratch root). See metal-hybrid-prefill-status-2026-06-28.md §2.
3 — Apple-Silicon / GPU / 27B-artifact gates
Each row needs a host this orchestrator does not have. Most rows remain not yet; any
row marked REACHED has an on-disk Mac witness recorded from the named node, never
re-measured here. The repro is the exact one command that re-confirms the gated half.
| # | Gated witness | Recorded prior result | Host needed | One-command repro |
|---|---|---|---|---|
| 1 | 27B greedy correctness parity (close the token-3 drift to PROVEN) | REFUTED — fak [248068,198,8160] vs llama.cpp [248068,198,90700] (near-tie argmax flip at step 2) |
M3 Pro + 27B GGUF + llama.cpp b9707 | go run ./cmd/qwen35check -gguf ~/.cache/fak-models/gguf/Qwen3.6-27B.q4_k_m.gguf -n 3 -topk 5 -ids <22 ChatML ids> -expect 248068,198,90700 (exits ≠0 today) |
| 2 | #71 Metal hybrid prefill — GPU f16 GEMM numerics | code landed; GPU-numerics parity unverified | M3 Pro, -tags fakmetal (macOS Metal toolchain) |
go test ./internal/model -tags fakmetal -run Qwen35Hybrid -count=1 |
| 3 | #63 on-device fak-Metal prefill tok/s (post-#1085, clean) | warm prefill 2.6 @P=27 / 7.3 @P=940 (cold first 0.5); bar 51.55 @pp22 | M3 Pro, -tags fakmetal, FAK_Q4K=1 FAK_METAL=1, no co-resident llama-server (36 GiB swap rule) |
FAK_Q4K=1 FAK_METAL=1 FAK_QPROFILE=1 fakchat -gguf Qwen3.6-27B.q4_k_m.gguf -tok <dir> ... → record [metalprof-hybrid …]; archived at experiments/qwen36/metal-fak-q4k-post1085-m3pro-20260628.json |
| 4 | #68 decode GEMV on-device parity | DONE in-thread (owner): TestMetalQ4KGemvMatchesCPU cosine 1.000000; TestMetalQ4KDecodeMatchesCPU decode == CPU [433 92 166 106] |
M3 Pro, -tags fakmetal |
go test ./internal/model -tags fakmetal -run 'MetalQ4K(Gemv\|Decode)' -count=1 |
| 5 | #70 q4_k device GEMM matmul-only split | code shipped; whole-path warm prefill 7.3 tok/s @P=940 (~7× under 51.55) | M3 Pro, -tags fakmetal, no co-resident llama-server |
go test ./internal/model -tags fakmetal -run MetalQ4K -count=1; then FAK_QPROFILE=1 pp22/long-prompt prefill |
| 6 | #69 zero-copy residency + residency-win measure | residency SHIPPED; newBufferWithBytesNoCopy upgrade + A/B win unmeasured |
M3 Pro, -tags fakmetal (also needs a (fak model)-lane FAK_METAL_REUPLOAD baseline toggle first) |
go test ./internal/model -tags fakmetal -run MetalQ4K -count=1 after the toggle lands |
| 7 | #67 end-to-end decode tok/s → 7.29 bar | clean decode 1.2 tok/s (ratio 0.16×, perf-gate FAIL is the expected fail-closed state) | M3 Pro, -tags fakmetal, no co-resident llama-server |
python tools/qwen36_perf_gate.py --metal --min-ratio 0.5 (exit 1 = recorded gap) |
| 8 | #65 GDN-recurrence on-device fraction | REACHED — CPU-hybrid decision witnessed; pp22 [metalprof-hybrid] measured total=6720.7 ms, gemm+roundtrip=6051.6 ms (90.0%), rest(recurrence/attn/norm)=669.1 ms (10.0%, upper bound on recurrence), so the #92 GPU-scan trigger did not fire |
M3 Pro, -tags fakmetal, captured on node-macos-a |
Witness: experiments/qwen36/metal-gdn-recurrence-m3pro-20260629.json; decision: experiments/qwen36/metal-gdn-recurrence-decision-2026-06-28.md |
The single one-command Mac gate that drives the still-open rows 2–7 in sequence is being
assembled at tools/qwen36_mac_parity_gate.sh (sibling agent, this campaign). Row 8 is
now independently witnessed by the #65 decision artifact above.
4 — Decode-progression reconciliation (0.1 → 0.9 → 1.2 vs 7.29)
There is one “fak Qwen3.6-27B decode” number, not three rivals: it is a measured
progression along three paths on one M3 Pro, all single-stream / batch=1.
QWEN36-PARITY-RESULTS.md is the source of record for the
full reconciliation table — do not duplicate it; this is the one-line summary:
| fak decode path | tok/s | what it measures |
|---|---|---|
| GGUF→Q8 cached (CPU) | 0.1 | one cached decode token through the GGUF→Q8 round-trip |
| resident-q4k microbench (CPU) | 0.9 | raw q4_k blocks resident, scalar-f32 GEMV (~9× the Q8 path) |
| resident-Q4_K Metal (GPU) | 1.2 | int8-SDOT Metal decode GEMV; bit-correct (cosine 1.0) but launch-bound |
vs llama.cpp-Metal 7.29 tok/s (the bar). Why fak sits ~6× under it is orchestration,
not arithmetic: each decode token runs ~336 separate Metal command-buffer GEMVs, each
~360 µs launch/sync-bound on top of ~98 µs of bandwidth-limited work. The kernels are
correct (GEMV cosine 1.000000 vs CPU; greedy token-parity). The proof the lever works:
BenchmarkMetalQ4KGemvBatch runs 64 GEMVs in one command buffer at 5.2× faster/GEMV
(89 GB/s, ~59% of device BW), projecting a one-command-buffer resident decode forward (#67)
to ~5.9 tok/s (→ ~8 with a kernel pass) — right at the 7.29 bar. Full diagnosis:
docs/notes/MAC-QWEN36-27B-Q4K-METAL-PERF-DIAGNOSIS-2026-06-26.md.
5 — The correctness token-3 drift (summary + pointer)
On the exact 22-token ChatML smoke prompt, greedy / temperature=0 / top_k=1:
| step | fak (in-kernel) | llama.cpp b9707 Metal | match |
|---|---|---|---|
| 0 | 248068 <think> |
248068 <think> |
✅ |
| 1 | 198 \n |
198 \n |
✅ |
| 2 | 8160 Here (logit 23.18; 2nd 90700 21.43) |
90700 Thinking (logprob −0.547; 2nd 8160 −0.945) |
❌ |
Both engines surface the same top-2 set {8160, 90700} and rank it oppositely — a
near-tie argmax flip (~1.75 logits on fak’s side, ~0.40 nats on llama.cpp’s), the
signature of accumulated float drift reaching the decision boundary by token 3, not a gross
error. The drift survives the move from GGUF→Q8 to native resident-q4k weights on both
engines, which disproves the “Q8 round-trip quant artifact” hypothesis; combined with
the tiny-fixture HF bit-exactness (§2) and Qwen3.5-0.8B f32 semantic correctness through the
same arch path, it localizes to a kernel-numerics divergence at 27B scale on the hybrid
GDN recurrence / mRoPE / partial-RoPE. Recorded as THEOREM 3 (REFUTED) in the proof
ledger. Pinned artifacts:
experiments/qwen36/native-gguf-q8-multitoken-parity-20260619.json (fak) and
experiments/qwen36/llamacpp-qwen36-multitoken-oracle-20260619.json (oracle). Sources:
experiments/qwen36/QWEN36-PARITY-AND-MEASUREMENT-STATUS-2026-06-20.md
§1, QWEN36-PARITY-RESULTS.md §”Token-3 drift RE-DIAGNOSED”.
Deeper root-cause investigation (which GDN/RoPE op compounds the error):
experiments/qwen36/token3-drift-investigation-2026-06-28.md (sibling agent, this campaign).
6 — The one-command Mac gate
tools/qwen36_mac_parity_gate.sh (sibling agent, this campaign) is the single entry point a
Mac verify node runs to re-confirm the gated half of §3 in one go — the -tags fakmetal
build + GPU numerics parity + the clean (no co-resident llama-server) FAK_QPROFILE tok/s
captures, against the recorded 51.55 / 7.29 bars. Until it is green on a witnessed commit,
rows 1–7 of §3 stay not yet.
Build-hole closure (2026-06-29, #1242). The prior ultracode dogfood (2026-06-28) left
exactly one hole: its build-sanity lane was policy-blocked in the subagent sandbox and
recorded as a SKIP, so the host-independent Go witness was never freshly built+run in
that wave. #1242 closes that hole with a real build:
TestQwen35HybridViaMMMatchesCPUTemplate was re-built and re-run green on a
windows/amd64 orchestrator at commit cf8af435 (an ancestor of origin/main), both with
the default build and CGO_ENABLED=0, and an independent multi-agent wave re-checked the
witness (build re-run, SKIP != PASS, commit ancestry, oracle ids, secret-shape scrub). Witness:
experiments/agent-live/qwen36-build-hole-closure-20260629T071148Z.json.
This closes a §2-class (host-independent) item; it does not move §3 — the Apple-Silicon
gate (tools/qwen36_mac_parity_gate.sh Arms 1–3 on a real M3 Pro) was not run here, so
rows 1–7 of §3 honestly stay not yet, gated on the M3 Pro verify node + the 27B GGUF +
llama.cpp b9707 that this orchestrator does not have. Row 8 is the later #65 decision witness,
recorded separately in
experiments/qwen36/metal-gdn-recurrence-m3pro-20260629.json.
7 — Adjacent axes — explicitly NOT the single-stream kernel numbers above
Two other Qwen3.6-27B figure families circulate; they are a different denominator and must not be quoted on the same line as the M3 Pro single-stream kernel rows:
- 8-GPU served throughput (single-stream ≈59–93 tok/s, batched peak ≈820–1085
completion tok/s) comes from SGLang-serves + fak-adjudicates on a datacenter GPU host,
not fak’s own M3 Pro engine — see
QWEN36-27B-GPU-SERVER-RESULTS.md. - AMD/Vulkan desktop (
QWEN36-AMD-VULKAN-RESULTS.md) proves the model loads and serves on an RX 7600, but llama.cpp logsfused Gated Delta Net (chunked) not supported, set to disabled— so its absolute throughput is not an apples-to-apples GDN bar. Likewise MLX is an invalid bar for this arch (its hybrid cache silently recomputes the full context every turn) — never used here; see…PARITY-AND-MEASUREMENT-STATUS…§2.
The single measured bar for the M3 Pro lane is llama.cpp b9707 Metal (-ngl 99):
prefill 51.55 tok/s, decode 7.29 tok/s, peak RSS ~24.5 GB; CPU-only (-ngl 0 -t 6):
20.12 / 6.48. (Source: QWEN36-PARITY-RESULTS.md.)
Rollup assembled 2026-06-28 on a win32 orchestrator (no Apple Silicon / no NVIDIA GPU / no 27B artifact). All speed/GPU/27B figures are recorded prior Mac/Vulkan-node witnesses, cited inline; none re-measured here.