GLM-5.2 “pure fak kernel” on a real GPU (8-GPU datacenter server) — witnessed results + the not-pure ledger (2026-06-21)

Goal (verbatim): prove end-to-end GPU on GPU server or H100 spot etc. running GLM-5.2 pure our fak kernel 100% as much as possible, and note anything that’s not pure that; benchmark and run and iterate deeply.

This doc is closed by witnesses the author did not write — nvidia-smi, nvcc, and go test -tags cuda exit codes captured on the live datacenter GPU node, and the on-device cosine /argmax verdicts the acceptance scripts emit — not by self-report. It supersedes nothing in GLM52-PURE-KERNEL-AND-AGENT-TURN-DEMOS-RESULTS-2026-06-21.md; it carries the on-GPU residual that doc explicitly handed off (“serving the real 753B is hardware-gated; the device numbers are handed off to a CUDA node”).

Current direction (see the staged plan, #917). This is a 2026-06-21 point-in-time witness. Its residual — “the pure fak kernel has no CPU-offload … serving the flagship at scale … out of scope here” — has since been superseded by progress: --cpu-offload-experts shipped and the full 466 GB model loads natively (see the staged plan and the 2026-06-25 native-serve note). Read the claims below as the snapshot they were taken at, not the current posture.

TL;DR (the honest split)

“Run GLM-5.2 on the pure fak kernel, on a GPU, end-to-end” is three different claims, and only the first is achievable today. This doc proves #1 on a real datacenter GPU and bounds #2/#3 exactly.

The pure fak CUDA kernel runs end-to-end on a real datacenter GPU (datacenter GPU). A full multi-layer prompt+decode forward runs on the device with every compute op being fak’s own kernel — Q8_0/Q4_K GEMM, fused flash attention, RMSNorm/RoPE/SwiGLU/Add/Argmax/KV-write — argmax-exact vs the CPU reference. In the Q8 path cuBLAS is never called. → §3, witnessed.
GLM-5.2’s architecture on that GPU path: was hard-blocked (#86); this session lands the first slice — witnessed on the datacenter GPU. Previously GLM-MoE-DSA panicked on any compute.Backend (CPU-only). Now GLM-5.2’s MoE/FFN experts + router + vocab head run on the pure fak GPU kernel (k_q8_gemm) via a new backendKernel; on the datacenter GPU the GLM-MoE-DSA forward matches the CPU Q8 forward at cosine = 1.000000, argmax-exact (TestCUDAGLMMoeDsaBackendForward). The remaining not-on-GPU piece is the DSA sparse-attention (learned-indexer top-k + sparse gather), still host-side — the next #86/#413 slice. → §4.
Serving the real 753B GLM-5.2 on this GPU server: does not fit, pure or not. 8-GPU datacenter server = 320 GB VRAM; INT4 GLM-5.2 ≈ 376 GB > 320 GB. The only way the full model “runs” here is CPU/host offload (llama.cpp, most weights in the GPU server’s ~1 TB RAM) — not the fak kernel and not pure-GPU. The pure fak kernel has no CPU-offload and no tensor-parallel/NCCL (a documented multi-month, 5-gap reshape), so a single model is capped at one datacenter GPU’s 40 GB. → §5.

§1 — The node + the access path

Node: the lab 8-GPU datacenter server (8× NVIDIA datacenter GPU, compute capability 8.0 / sm_80), 256 logical cores, ~1007 GB host RAM. GPUs idle except GPU0, which holds ~6 GB — a llama.cpp llama-server (the CPU-offloaded GLM path; see §5). CUDA 12.8 (/usr/local/cuda), go1.26.0.
Reach: only via the private control bridge (private lab tooling, pipe mode — the readback fix from the prior handoff is live; selftest round-trips an echo OK). The build host is win32 with no CUDA toolkit / no GPU, so the device execution below is the explicit residual it cannot run.
Why a fresh clone on the node: the node’s /srv/fleet/fak is an older rsync whose cuda_kernels.cu predates the Q8/Q4K/flash kernels. private pure-kernel runner clones the pushed public HEAD onto the node and builds it for sm_80, so the device runs exactly the code on origin/main.

§2 — What “pure fak kernel” means here (the not-pure ledger)

The CUDA backend (internal/compute/cuda.go + cuda_kernels.cu, built only under -tags cuda) routes each op as follows. This is the central “what is / isn’t pure” accounting:

Op	Pure fak kernel?	Implementation
MatMul / BatchedMatMul, Q8_0	✅ pure	`k_q8_quant_act` + `k_q8_gemm` (on-device int8 act-quant, integer per-block dot)
MatMul / BatchedMatMul, Q4_K	✅ pure	`k_q4k_gemm` (dequant fused into the tile off resident super-block bytes)
MatMul, AWQ 4-bit	✅ pure	`k_awq_gemm` / `k_awq_gemv`
Attention (decode)	✅ pure	`k_flash_attention` (fused online-softmax; no scores row materialized)
RMSNorm / RoPE / SwiGLU / Add / AddBias	✅ pure	`k_rmsnorm` / `k_rope` / `k_swiglu` / `k_add` / `k_add_bias`
Argmax / KV append	✅ pure	`k_argmax` / `k_copyrow` (graph-patchable scalar-offset KV write)
CUDA graph capture/replay	✅ pure	`fcuda_graph_*` (instantiate-once / ExecUpdate-many)
MatMul, F32	❌ cuBLAS	`cublasSgemm` (NVIDIA BLAS; different reduction order → Approx)
MatMul, F16 (tensor-core HGEMM)	❌ cuBLAS	`cublasGemmEx` (NVIDIA BLAS, tensor cores)
device alloc / H2D-D2H / memcpy / launch	❌ NVIDIA	CUDA runtime + driver (unavoidable for any GPU program)

Consequence: a Q8_0-quantized decode on -backend cuda -quant calls ZERO cuBLAS — every GEMM is k_q8_gemm, attention is k_flash_attention, all elementwise/reduction ops are fak kernels. The only non-fak code on that path is the CUDA runtime/driver itself. That is the strongest honest meaning of “100% pure fak kernel on the GPU,” and it is what §3 witnesses. (The F32 and F16 paths lean on cuBLAS; they are not “pure” in the kernel sense.)

§3 — Pure fak kernel on the datacenter GPU: witnessed (live run, 2026-06-21)

Run via private pure-kernel runner on the node (clone origin/main → nvcc -arch=sm_80 → -tags cuda witnesses). Node: NVIDIA datacenter GPU, tier sm_80, CUDA 12.8, go1.26.0.

Build: internal/compute/build_cuda.sh build (nvcc sm_80 → libfakcuda.a; go build -tags cuda ./internal/compute/) → rc=0. The pure-Go-plus-cgo CUDA backend compiles and links on the datacenter GPU.

Pure-kernel device witnesses — PASS in isolation (every op is fak’s own kernel):

Witness	Result (datacenter GPU, sm_80)	Kernel
`TestCUDAForwardMatchesRef` (multi-layer decode forward, 6 prompt + 8 greedy)	PASS — argmax-exact, cosine = 1.00000000 (graphs off and `FAK_CUDA_GRAPH=1`)	k_q… + k_flash_attention + k_rmsnorm/rope/swiglu/argmax
`TestCUDAFlashAttentionMatchesRef` (MHA/GQA/MQA)	PASS — cosine = 1.0 (maxAbs 2.4e-7…4.2e-7) vs cpuref AND vs naive	`k_flash_attention`
`TestCUDAQ8MatMulApproxMatchesRef` (decode GEMV)	PASS — cosine 0.99999970, argmax-exact	`k_q8_quant_act`+`k_q8_gemm`
`TestCUDAQ8BatchedMatMulApproxMatchesRef` (P=8 prefill)	PASS — cosine 0.99999969	`k_q8_gemm`
`TestCUDAQ4KBatchedMatMulApproxMatchesRef` (P=8 prefill)	PASS — cosine 1.00000000	`k_q4k_gemm`
`TestCUDAQuantVRAMWitness` (weight resident at int8/int4 size)	PASS	upload path

The forward witness is the headline: a full multi-layer decode forward runs end-to-end on the datacenter GPU with every compute op being a fak kernel (no cuBLAS — the synthetic weights load as Q8 / the forward routes through k_q8_gemm + k_flash_attention), bit-faithful to the CPU reference (argmax-exact, cosine 1.0).

Two real on-hardware findings (failures, not hidden):

TestCUDAQ4KMatMulApproxMatchesRef (Q4_K GEMV, P=1) FAILS its argmax-exact gate. Root cause: the test builds an activation with a narrow dominant-channel margin (alignActToRow) and demands argmax-exact, but Q4_K’s 4-bit reconstruction error (maxAbs ≈ 1.2e-3) flips that narrow argmax on sm_80 — while the Q4_K batched path (cosine-only gate) passes at cosine 1.0 with the same kernel. So the kernel is correct; the GEMV test’s argmax-exact gate is too tight for a 4-bit format. Fix = widen the constructed margin or drop argmax-exact for Q4_K GEMV (cosine ≥ 0.995 is the honest gate). Fileable as a test-fragility bug.
The combined -tags cuda suite (build_cuda.sh test, the FAK_CUDA_GRAPH=1 graph-capture pass) panics: index out of range [1] with length 1 at cuda.go:574 (w.Shape[1]) — a 1-D-shaped weight reaches MatMul only when the whole suite runs together (the forward witness passes cleanly in isolation and graphs-off). A cross-test/graph-path state bug, not a kernel correctness defect. Fileable; needs datacenter GPU iteration to fix.

End-to-end decode throughput (private pure-kernel benchmark runner, SmolLM2-135M Q8 on the datacenter GPU, the pure k_q8_gemm + k_flash_attention decode path — the closest honest “real model generating tokens on the pure fak GPU kernel,” since GLM-DSA can’t take the backend, §4):

modelbench -backend cuda -lean -hf SmolLM2-135M -decode-steps 128   (A100, tier sm_80, GOMAXPROCS=256)
engine: fak-in-kernel via compute HAL backend "cuda"   backend.selected=cuda  backend.tier=sm_80
prefill P=16 : 118.6 ms (134.9 tok/s)
prefill P=64 : 396.0 ms (161.6 tok/s)
prefill P=256: 2444.1 ms (104.7 tok/s)
decode       :   7.8 ms/tok (127.8 tok/s)

So the pure fak kernel generates tokens end-to-end on the datacenter GPU at 127.8 tok/s decode on a real checkpoint, every compute op a fak kernel (Q8 weights → k_q8_gemm; attention → k_flash_attention; RMSNorm/RoPE/SwiGLU/argmax → fak kernels) — no cuBLAS on the Q8 path. (Getting here required the §3.5 fixes; the lean-Q8 upload bug had to be fixed before this path ran at all.)

§3.5 — Iterations on the datacenter GPU (bugs found + fixed, real run loop)

Driving the pure-kernel path to a green end-to-end GPU decode surfaced and fixed real bugs (each a commit on origin/main, each re-verified on the datacenter GPU):

go aborted: GOCACHE not defined — the setsid-detached worker runs under the control bridge’s non-interactive shell, which has no $HOME. Fix: pin HOME/GOCACHE/GOPATH in the runner (0a370c5).
go run -tags cuda link error (collect2: ld returned 1) — the bench’s go run lacked the CUDA cgo link flags (-L$CUDA_HOME/lib64 for -lcudart/-lcublas). Fix: export CGO_CFLAGS/CGO_LDFLAGS/LD_LIBRARY_PATH (cae5be9).
Pure-Q8 e2e decode panicked at the first weight H2D — cuda.Upload(_, Q8_0) only accepted F32 host data to narrow on-device, but the memory-lean load hands an already-quantized Q8_0 host tensor (codes+scales; the f32 was dropped at load) via the HAL’s weightHALQ8. So modelbench -backend cuda -lean — the pure k_q8_gemm decode path — crashed even though the Q8 GEMM kernel itself passes. Fix: uploadQ8Resident copies the int8 codes + per-block f32 scales resident directly (same layout as the f32-narrowing path, so k_q8_gemm consumes it unchanged), additive and leaving the witness path untouched (c009737). This is also a prerequisite for any future GLM-DSA-on-backend — the GLM-DSA Q8 weights would upload through the same seam.

Plus the two witness findings in §3 (Q4_K GEMV argmax-gate too tight for 4-bit; combined-suite graph-path panic) — filed, not hidden.

§4 — GLM-5.2 on the GPU path: #86 was a hard block; this session lands the first slice (witnessed on the datacenter GPU)

Before this session: Session.requireGLMDsaSession() panicked on any s.Backend != nil — GLM-MoE-DSA refused to run on any accelerated compute.Backend; CPU-resident only (issue #86; docs/notes/model-arch-seam-status-487.md GLM-5.2 row: accel decode “no — #86”).

This session (93119eb, on-device cf9d9a1): the guard now PERMITS a backend, and GLM-5.2’s dense GEMMs route through it — the MoE/FFN experts + router (a new backendKernel swapped into decodeBandGLMDsa’s matKernel) and the vocab head (glmDsaHead). With a lean (Q8-resident) model on the cuda backend those GEMMs run on k_q8_gemm — the pure fak GPU kernel. The MoE/FFN is the bulk of GLM-5.2’s parameters, so most of its decode FLOPs now execute on the GPU.

Witnessed:

TestGLMMoeDsaBackendGEMMMatchesCPU (cpu-ref): a GLM-MoE-DSA Prefill with MoE/FFN+head GEMMs routed through the backend is bit-exact (max|Δ| = 0), argmax-exact vs the all-host CPU forward — the routing is correct; the 23 prior GLM-DSA witnesses stay green.
TestCUDAGLMMoeDsaBackendForward on the datacenter GPU (sm_80): PASS — GLM-MoE-DSA MoE/FFN+head GEMMs on k_q8_gemm, cosine = 1.000000, argmax cpu=40 cuda=40 (argmax-exact) vs the CPU Q8 forward. This is GLM-5.2’s own architecture running its dense compute on the pure fak GPU kernel, on real datacenter hardware.

What is still NOT on the GPU (the honest residual): the DSA-specific work stays host-side in this slice — the learned-indexer top-k scoring (glmDsaIndexStep), the sparse attention gather/softmax/ΣwV (glmDsaAttendCached), and the DSA KV cache. A fully GPU GLM-DSA needs a fused sparse-attention CUDA kernel + device DSA-KV — the next slice of #86/#413. So today: GLM-5.2’s MoE/FFN/head = pure fak GPU kernel; DSA attention = host. (The GLM arch also remains green end-to-end on CPU — the prior doc’s §1.)

§5 — The real 753B: why it doesn’t fit, and what the ~6 GB on GPU0 actually is

753B at INT4 ≈ 376 GB > 320 GB total datacenter GPU VRAM (FP8 ≈ 750 GB). It cannot be GPU-resident on this GPU server.
The resident ~6 GB on GPU0 is a llama.cpp llama-server — i.e. CPU/host offload: a few layers on the GPU, the bulk in the GPU server’s ~1 TB host RAM. That is doubly not-pure (not fak; not pure-GPU) and CPU-memory-bandwidth bound.
The pure fak kernel has no CPU-offload and no tensor-parallel/NCCL path — the documented 5-gap, multi-month reshape (sharded+quant load, TP backend seam + NCCL, paged device KV, batched decode, fused sparse-attention kernel). The partition/pipeline/handoff seams exist and are bit-exact, but the NCCL wire + fused sparse kernel that 753B needs are not built. So serving the flagship at scale on the pure kernel is not achievable in a session, and is honestly out of scope here.

What is proven vs not (labeled, not hidden)

Proven on the datacenter GPU (§3), live: the pure fak CUDA kernel builds for sm_80 and runs a full multi-layer decode forward end-to-end on real datacenter hardware — argmax-exact, cosine = 1.0, with zero cuBLAS in the Q8 path (every op is k_q8_gemm / k_flash_attention / k_rmsnorm / …). Flash attention, Q8 GEMV+prefill, and Q4_K prefill all match the CPU reference on the device, and SmolLM2-135M Q8 decodes end-to-end at 127.8 tok/s on the pure path.
GLM-5.2 specifically, on the datacenter GPU (§4), live: GLM-MoE-DSA’s MoE/FFN experts + router + vocab head now run on the pure fak GPU kernel (k_q8_gemm) — TestCUDAGLMMoeDsaBackendForward: cosine = 1.000000, argmax-exact vs the CPU Q8 forward. The DSA sparse-attention stays host-side (the labeled residual). This is the first slice of #86, landed + witnessed this session.
Found on the datacenter GPU (labeled bugs, filed): Q4_K GEMV argmax-exact gate too tight for 4-bit (kernel correct; test fixture); a combined-suite graph-path panic (1-D weight → MatMul). Neither is a kernel-correctness defect — the isolated kernels all pass.
Not proven / out of scope (labeled): GLM-5.2’s DSA sparse-attention on the GPU (needs a fused sparse-attention CUDA kernel + device DSA-KV — #86/#413 next slice); HF numeric DSA parity (#474/#413, oracle-gated); real 753B serving (VRAM: INT4 ≈ 376 GB > 320 GB; plus the NCCL/offload reshape). Each is bounded exactly above — none is faked.

Reproduce: bash private pure-kernel runner on an sm_80 CUDA node (or via the control bridge: ship it, exec 'bash <private-scratch>/pure-kernel-run.sh', poll <private-scratch>/run.log + <private-scratch>/DONE.<rc>).

GLM-5.2 pure fak kernel on an 8-GPU datacenter server (2026-06-21)

On-device datacenter GPU witnesses for the pure fak CUDA kernel running end-to-end, the first GLM-MoE GPU slice, and the not-pure ledger bounding the 753B serving gap.