Native Q8_0 / Q4_K device GEMM, recorded on a datacenter sm_80 GPU node (2026-06-24)

What this is: the first time fak’s native quantized device-GEMM acceptance (the #485 -tags cuda witness) has been run to a recorded number on a real GPU node. The native-753B staged plan (native-753b-track-staged-plan.md) names this as a Pillar-2 gap in two places — “the CUDA Q4_K/Q8_0 cosines have never been run to a recorded number on a GPU node” and the honesty-ledger “Scale gap” — and this note closes it. The numbers are grounded in the go test exit code and the run log of tools/run_485_acceptance_on_gpu.sh, not a self-report.

All runs are on a fresh, idle 8× datacenter GPU (sm_80, compute 8.0), CUDA 12.8 (V12.8.61), ~2 TB host RAM node, GPU 0, at origin/main HEAD 26fe933, Go 1.26.4. The kernels were built with nvcc -arch=sm_80 into libfakcuda.a; the test backend self-reported device=cuda tier=sm_80 class=approx.

1. The per-dtype cosine gates — all PASS, argmax-exact

Each device quantized GEMM is held to its own recorded cosine floor against the cpu-ref f32 Reference (the Approx class — bit-identity is deliberately off the table; the device quantizes). The weight stays NARROW in VRAM — int8 codes for Q8_0, raw Q4_K super-block bytes for Q4_K — and the GEMM consumes it directly, no dequant-to-f32 round trip (Q8_0 quantizes the activation on-device; Q4_K fuses the dequant into the GEMM tile).

Witness	Shape	cosine	gate	max|Δ|	verdict
TestCUDAQ8MatMulApproxMatchesRef (decode GEMV, P=1)	[320,256]	0.99999980	0.999	6.91e-03	PASS, argmax-exact
TestCUDAQ8BatchedMatMulApproxMatchesRef (prefill GEMM, P=8)	[320,256]	0.99999969	0.999	3.64e-02	PASS
TestCUDAQ4KMatMulApproxMatchesRef (decode GEMV, P=1)	[320,256]	1.00000000	0.995	5.47e-02	PASS, argmax-exact
TestCUDAQ4KBatchedMatMulApproxMatchesRef (prefill GEMM, P=8)	[320,256]	1.00000000	0.995	1.22e-03	PASS

Two facts worth reading carefully:

• Q8_0 sits a hair below 1.0 because the gate measures real quantization error: the weight is narrowed to int8 codes + per-block f32 scales at H2D and the activation is quantized to int8 on-device, so the cosine is the genuine Q8 weight+activation quant error against the unquantized f32 reference. It clears its 0.999 floor with margin.
• Q4_K reads as exactly 1.0 because that gate isolates the device dequant-fused tile arithmetic: its reference is an f32 dequant of the same super-block bytes, so a correct getScaleMinK4 6-bit unpack and w = d·scale·code − dmin·min reconstruction on the device reproduces the host f32 dequant to the cosine’s printed precision. This is the exact trap the comment in cuda_quant_test.go calls out — a wrong GLSL/CUDA port of the k-quant geometry silently collapses the cosine — and on this node the port is faithful. (The looser 0.995 floor exists for the full-model true-f32 → Q4_K reconstruction residual, which this isolated gate does not exercise.)

2. VRAM witness — the weight stays int8/int4-sized

Read straight off the device buffers (residentWeightBytes), not a host estimate, for a [512,256] weight:

• f32: 524 288 B
• Q8_0: 147 456 B — 3.56× smaller (int8 codes + per-block(32) f32 scales)
• Q4_K: 73 728 B — 7.11× smaller (raw super-block bytes, ≈0.56 byte/elem)

This is the whole point for 753B: the resident weight is a small fraction of the f32 bytes a dequant-to-f32 upload would have paid.

3. Throughput — honestly slower in raw FLOPs; the win is footprint

The same harness ran the quantized GEMM benchmarks beside the F32 SGEMM baseline (a 4096×4096 weight, P=512 prefill tile):

Kernel	ms / GEMM	GFLOP/s	vs F32
F32 SGEMM (baseline)	2.867	5992.3	1.00×
Q8_0 (resident int8, on-device act quant)	34.655	495.7	0.08×
Q4_K (resident super-blocks, dequant fused)	81.380	211.1	0.04×

Read this straight: the quantized kernels are slower than f32 SGEMM in raw FLOP/s. They are correctness-first kernels — a dequant-in-the-inner-loop GEMM, not a tiled, tensor-core path — so they trade arithmetic throughput for a small resident weight. The win they buy is the VRAM/bandwidth footprint in §2, which is what lets a model that does not fit in f32 fit at all; it is not a speed win yet. Closing the speed gap (tiling + tensor cores for the quantized path) is separate, later work and is not claimed here.

4. What this retires, and what is still open

Retired (Pillar 2, “run the gates to a recorded number”): the native quant device GEMM now has a recorded, reproducible cosine on real sm_80 hardware. The staged plan’s P2 Metal HAL + CUDA witness milestone had two halves; the CUDA-witness half is done and pinned to the numbers above. The honesty-ledger “Scale gap” line — “the CUDA quant cosines have never been recorded on hardware” — no longer holds.

Still open (unchanged, do not over-claim):

• Vulkan Q4_K GEMM (AMD path is Q8_0-only) and the Metal HAL Q4_K exposure — the other half of the P2 milestone — still need a node each.
• Full-model mixed-precision forward: these gates isolate single GEMMs on synthetic super-blocks. The end-to-end Q4_K + Q8_0/f32 glm_moe_dsa forward witness (P2 “Full-model quant forward”) still depends on the Pillar-1 GGUF load path.
• 753B serving remains the labeled multi-month integration: real GGUF → mixed-precision device GEMM → multi-GPU TP/EP → tiered offload. This note moves one device-GEMM rung from “type-checks on the build host” to “recorded on the GPU”; it does not move the integration.

See native-753b-track-staged-plan.md for the full pillar map and the dependency-ordered milestones, and GLM52-REAL-ORACLE-GENERATION-ON-PURE-FAK-2026-06-23.md for the matching real-checkpoint generation witness.