Qwen2.5-7B-Instruct — first large dense run in fak + llama.cpp parity (rung 2)
Purpose. The
GPU-MODEL-PICKtarget size, run in fak on this M3 Pro (CPU Q8 /-lean, since the discrete-GPU story is the Windows RTX box, not this Mac). Proves the dense path scales to 7B and gives the first real fak-vs-llama.cpp throughput comparison at a useful size. This is the rung that establishes the honest raw-throughput gap fak’s reuse stack has to overcome.
Host: node-macos-a / Apple M3 Pro (6P+6E, 18-core GPU, 36 GB) / Go 1.23.1.
Date: 2026-06-19.
Quant: Q8_0 (both engines — apples-to-apples).
fak in-kernel (CPU Q8, NEON)
go run ./cmd/modelbench -hf ~/.cache/fak-models/qwen2.5-7b-instruct -lean -quant (artifact:
experiments/model-ladder/modelbench-qwen25-7b-q8.json, load ~73 s):
| prefill P16 | prefill P64 | prefill P256 | decode |
|---|---|---|---|
| 15.4 tok/s | 16.0 tok/s | 16.1 tok/s | 8.7 tok/s (115.2 ms/tok) |
One Qwen2.5-7B fak decode number, and it is in the Authority (#123). The pinned figure is the 8.7 tok/s (115.2 ms/tok) above — artifact
experiments/model-ladder/modelbench-qwen25-7b-q8.json, indexed inBENCHMARK-AUTHORITY.md(the single source of truth). A lower ~8.2 tok/s decode figure for the same engine/model/box has circulated from a separatefak-native-chat run; it is not a second authoritative measurement. Read 8.7 tok/s as the number of record and ~8.2 tok/s as a different (earlier/contended) run superseded by it — so the two figures are reconciled here rather than left to circulate unpinned.
llama.cpp b9707 reference (Metal, full offload)
llama-completion -m qwen2.5-7b-instruct-q8_0.gguf -ngl 99 -t 6 -c 4096 -n 32 (load 3.16 s):
| prefill | decode |
|---|---|
| 192.92 tok/s | 17.6 tok/s |
fak vs llama.cpp (the honest throughput gap)
| phase | fak (CPU Q8) | llama.cpp (Metal Q8) | fak ratio |
|---|---|---|---|
| prefill | 16.1 tok/s | 192.9 tok/s | 0.083× |
| decode | 8.7 tok/s | 17.27 tok/s | 0.50× |
Consistent with the repo’s standing honesty bound (SESSION-VALUE-STACK-RESULTS.md,
M3-LLAMACPP-RESULTS.md: dense fak single-stream ≈0.46× decode / ≈0.15× prefill vs llama.cpp build 8200 on
M3 Q8). Prefill is compute-bound (Metal GPU gives llama.cpp ~12× here); decode is bandwidth-bound
so the gap closes to 2×. fak does not beat llama.cpp on raw tok/s — the moat is the reuse
stack + in-kernel integration, not single-stream throughput. The prefill half is the open Metal
GEMM lane (internal/metalgemm, -tags fakmetal).
Parity — greedy agreement (the correctness gate)
Same ChatML prompt, greedy (temp 0), Q8_0 both engines:
| engine | output |
|---|---|
| fak | 2+2 is 4. |
| llama.cpp | 2+2 is 4. |
Full 7-token greedy match. First-token (and whole-short-generation) parity on the 7B Q8 — the
dense forward path is numerically faithful at this size, not just at the 135M/1.5B anchor sizes.
This extends the parity line (fak/experiments/parity/PARITY.md) one rung up.
Bottom line
- 7B runs in fak on this Mac, fits in 36 GB unified with headroom (Q8 weights ~7.6 GB + KV + compute), load ~73 s cold.
- fak is 0.083× prefill / 0.50× decode vs llama.cpp Metal — the expected honest gap. The reuse
stack (
radixbench/sessionbench) is what makes fak competitive on agent workloads despite this single-stream gap, not raw tok/s. - Greedy parity holds —
2+2 is 4.matches token-for-token. The 7B is a legitimate large dense rung, not just a fit exercise.
Implements rung 2 of PLAN-model-ladder-qwen36-2026-06-19.md. fak artifact:
experiments/model-ladder/modelbench-qwen25-7b-q8.json. Dated 2026-06-19.