GLM-5.2 in fak’s own kernel on the sm_80 datacenter node — load works, load-speed is the open problem (2026-06-25)

Status snapshot for the next session. fak’s own in-kernel engine (not llama.cpp) now loads the full 466 GB GLM-5.2 (glm_moe_dsa, unsloth UD-Q4_K_M, 11 shards) on the 8× sm_80 datacenter GPU (80GB) box and binds /v1/*. Every loader/fit/inference gap that blocked it is fixed and shipped. The one open item is load time (~100 min) — and the cause is now precisely diagnosed.

Update 2026-06-26 — load-speed levers shipped (on-box re-measure is the open witness)

The two diagnosed causes below are now fixed in code on origin/main; the remaining work is to re-measure the 466 GB load on the box.

Update 2026-06-27 — WITNESSED on GPU server: 466 GB load ~100 min → 150 s (3.04 GB/s)

Run on GPU server (8-GPU datacenter server, sm_80), origin/main 6d727be7, the staged 466 GB UD-Q4_K_M checkpoint on local NVMe, via tools/glm52_load_witness.sh (fak serve --gguf <shard1> --backend cuda --cpu-offload-experts --context-budget-tokens 8192, FAK_GGUF_LOAD_WORKERS=64):

BUILD_OK
LOAD_READY 150s (2m30s)  under_10min=YES        # fak_model_load_duration_seconds = 145.5
fak: loading model 100% (1809/1809 tensors, 433.8 GB, 2m23s elapsed, 3.04 GB/s)
fak: load-path breakdown (resident = raw bytes, no dequant; dequant = f32 round-trip):
fak:   Q4_K  expert  resident=38912 (256.5 GB)  dequant=0 (0.0 GB)
fak:   Q5_K  expert  resident=18688 (150.6 GB)  dequant=0 (0.0 GB)
fak:   Q6_K  expert  resident=768   (7.4 GB)    dequant=0 (0.0 GB)
fak:   Q8_0  dense   resident=0                 dequant=712 (16.9 GB)
fak:   F32   dense   resident=0                 dequant=706 (0.5 GB)

So every routed-expert quant type loads resident with zero f32 round-trip — the 158 GB of Q5_K/Q6_K experts that used to take the slow path (the ~100-min cause) now copy raw, and the load is I/O-bound at NVMe speed. Only the small dense set (~18 GB Q8/F32) dequants. A warm-cache re-run landed in 136 s (3.75 GB/s). The <10-min load target is met.

End-to-end RUNS (decode proven, but slow). The served /v1/chat/completions completed on fak’s own kernel — the serve log shows inkernel_chat model=glm-5.2 q4k=true prompt=33tok prefill=33tok/144.49s/0.2tok/s decode=2tok/8.77s/0.2tok/s: prefill AND decode produce tokens through the resident Q4_K/Q5_K/Q6_K experts. (A short max_tokens smoke returns empty “content” because GLM-5.2 spends the first tokens on reasoning_content — a harness caveat, not a serving failure.)

Remaining (separate axis): per-token throughput is ~0.2 tok/s — the 753B experts run on the host CPU under --cpu-offload-experts via the correctness-first scalar k-quant GEMV. The LOAD is the proven win here; decode throughput is the next perf lever (an int8-SDOT k-quant GEMV like q4k already has for Q4_K, extended to Q5_K/Q6_K, and/or paging experts to the device).

Original open-witness note (now closed by the run above)

Re-run the self-staging serve on GPU server and confirm the load is < 10 min with every routed-expert quant type on the resident path. The expected shape: fak_model_load_path_tensors{...,path="dequant"} ≈ 0 for the expert quant types, and the stderr load-path summary shows resident for Q4_K and Q5/6_K. The remaining serial cost is the small dense set (attention/router/shared/embed/lm_head); the 417 GB expert bulk is now a raw copy.

What is fixed and shipped (origin/main)

GLM-5.2 config (from cfgdiag): NumLayers=79 Hidden=6144 NumHeads=64 IndexNHeads=32 IndexHeadDim=128 IndexTopK=2048 QLoraRank=2048 KVLoraRank=512 FirstKDenseReplace=3.

The open problem: ~100 min load is CPU-bound in the loader, NOT disk I/O

Measured, not assumed:

Leading hypothesis (verify first next session)

UD-Q4K_M is unsloth’s _dynamic mixed quant — the experts are not all Q4_K. The raw-resident expert split only fires for info.Type == TensorQ4_K; any Q6_K / Q5_K expert tensors fall back to the slow f32 dequant→split. Since the experts are the 417 GB bulk, mixed-quant experts would keep most of the load on the slow path.

Confirm with: gguf-dump --no-tensor-data <shard5>.gguf | grep exps (gguf-dump is at /usr/local/bin on the box). If mixed, extend the raw-resident split to the other K-quants (add Q6_K=210 B/256w and Q5_K=176 B/256w super-block sizes + AddResidentQ6K/Q5K, or a generic raw-resident-by-type splitter keyed on info.Type). Second lever: confirm the dequant is actually parallel across the 261 threads — if it’s serial per tensor, thread it.

Next steps (priority order)

  1. Confirm the expert quant histogram (gguf-dump).
  2. Extend raw-resident split to the mixed K-quants (or fix loader threading) → target ≤10 min load.
  3. Serve from the NVMe path (/mnt/sglang_dv3/glm52-q4/...-00001-of-00011.gguf), not /projects (NFS). With the MLA dims fixed, the smoke decode should now pass.
  4. Run the e2e (tools/private GLM-5.2 e2e runner): the #413 serving witness + fak guard --provider openai --base-url http://127.0.0.1:8000/v1 -- claude against the fak-native endpoint. Pass --model glm-5.2 to label /v1/models (today it shows the --model default mock; the in-kernel planner IS active — the tokenizer loads fine, FromGGML ok, 154880 tokens / 321649 merges, pre=glm4).

Operational notes