GPU-server GLM-5.2 vs vLLM — agentic benchmark plan

Status: pending measurement. This document carries commands and gates, not results. No GLM-5.2/vLLM resolve-rate, tax, or throughput number may be quoted until the JSON artifacts named below are produced on a real serving node and linked from BENCHMARK-AUTHORITY.md.

What We Are Comparing

There are two different comparisons, and the reports must keep them separate:

axis	fair comparison	required artifact
Gateway tax / completion	GLM-5.2 served by raw vLLM vs the same raw vLLM endpoint behind `fak serve`	`experiments/vllm/glm52-agentic-battery/raw-vllm-vs-fak-gateway-contract.json`; `experiments/vllm/adjudication-tax-witness.json`; SWE-bench `COMPARE-PREFLIGHT.json` + `compare.json`
Native fak kernel vs stock engines	same GLM-family checkpoint, same hardware, same precision/context/batch, fak native vs vLLM/SGLang/llama.cpp	future B2/B3 artifacts; the current `glmdsatput` numbers are synthetic kernel-cost only

Do not compare fak’s synthetic GLM kernel tok/s directly to a full-size vLLM serving tok/s. The live vLLM run is full external-engine serving; the current native fak GLM numbers are reduced-scale device-kernel cost.

Run Matrix

rung	benchmark	question answered	command / artifact
0	contract	is the raw-vLLM/fak-gateway comparison identity pinned before any result claim?	`tools/glm52_vllm_agentic_battery.py --contract-only` -> `raw-vllm-vs-fak-gateway-contract.json`
1	readiness	can this node serve GLM-5.2 with vLLM at all?	`tools/glm52_serve_preflight.py` -> `preflight.json`
2	serving witness	does GLM-5.2 answer direct, through fak, and quarantine a poisoned tool result?	`tools/glm52_serving_witness.py` -> `experiments/glm52/full-size-serving-witness.json`
3	vLLM adjudication tax	what latency/decode tax does `fak serve` add over raw vLLM?	`tools/vllm_tax_witness.py` -> `experiments/vllm/adjudication-tax-witness.json`
4	SWE-bench Verified 20	does the agent finish/resolve the same 20 Verified tasks raw-vLLM vs fak-gateway?	private compare runner preflight, then the 20-task run -> `COMPARE-PREFLIGHT.json` + `compare.json` + `COMPARE.md` + `DONE.rc`
5	fak-native agentic floors	what are the deterministic turn-tax/session/fanout/cache-reuse floors independent of live model variance?	`fak swebench compare`, `turntax`, `sessionbench`, `fanbench`, `radixbench` artifacts

Rungs 0-4 are the GLM-5.2/vLLM live-serving comparison. Rung 5 is the fak-native agentic mechanism series; it explains what fak should move, but it is not a raw vLLM head-to-head unless the workload also drives the same served endpoint.

Reproduce On A GLM-5.2-Capable Node

Run this on a Hopper/Blackwell-class serving node. An Ampere sm_80 node is expected to fail the stock vLLM GLM-5.2 preflight because stock DSA kernels require sm_90+.

# The fak-native SWE-bench floor needs bench's official difficulty map or a full
# SWE-bench Verified dataset export. The live resolve-rate arm below still grades
# through the official harness; this file is for the deterministic floor geometry.
: "${FAK_SWEBENCH_DIFFICULTY:?set FAK_SWEBENCH_DIFFICULTY to swebench_verified_difficulty.json}"

# Optional: write the auditable command manifest and check any existing artifacts.
# The generated run.sh contains the same commands below; run it instead of
# copy/pasting the rest of this block if you want a single runner. It runs the
# strict artifact gate at the end and fails if any result is missing or stale.
# On success, that final gate writes FINAL-CHECK.md and a guarded
# BENCHMARK-AUTHORITY-DRAFT.md snippet; no draft is written while pending.
python tools/glm52_vllm_agentic_battery.py \
  --out experiments/vllm/glm52-agentic-battery/manifest.json \
  --markdown experiments/vllm/glm52-agentic-battery/MANIFEST.md \
  --script experiments/vllm/glm52-agentic-battery/run.sh \
  --run-contract experiments/vllm/glm52-agentic-battery/raw-vllm-vs-fak-gateway-contract.json \
  --swebench-difficulty "$FAK_SWEBENCH_DIFFICULTY" \
  --allow-pending

# Before starting vLLM, set GLM52_TOOL_CALL_PARSER to the parser name required
# by the vLLM build/model card. This guard refuses to continue if it is unset.
: "${GLM52_TOOL_CALL_PARSER:?set GLM52_TOOL_CALL_PARSER to the vLLM parser name}"

# 0. Preflight the node and fail closed if GLM-5.2/vLLM is not viable.
python tools/glm52_serve_preflight.py \
  --engine vllm --quant fp8 --require-ready \
  --out experiments/vllm/glm52-vllm-preflight.json \
  --markdown experiments/vllm/glm52-vllm-preflight.md

# 1. Start raw vLLM for the GLM-5.2 fp8 checkpoint. Add the model-specific parser flags
# required by the vLLM build/model card via ENGINE_ARGS.
ENGINE=vllm SERVED_NAME=glm-5.2 PORT=8000 \
  ENGINE_ARGS="--enable-auto-tool-choice --tool-call-parser ${GLM52_TOOL_CALL_PARSER}" \
  bash tools/glm52_sglang_vllm_serve.sh

# 2. Prove direct + fak-gateway + quarantine behavior.
python tools/glm52_serving_witness.py \
  --base-url http://127.0.0.1:8000/v1 \
  --model glm-5.2 \
  --engine-cache-engine vllm \
  --context-length 131072 \
  --out experiments/glm52/full-size-serving-witness.json \
  --markdown experiments/glm52/full-size-serving-witness.md

# 3. Measure the fak-over-vLLM gateway tax.
python tools/vllm_tax_witness.py \
  --base-url http://127.0.0.1:8000/v1 \
  --model glm-5.2 \
  --count 8 \
  --record \
  --out experiments/vllm/adjudication-tax-witness.json \
  --markdown experiments/vllm/adjudication-tax-witness.md

# 4a. Fail fast before the long SWE-bench Verified comparison. This checks the
# runner/grader install, raw endpoint, GPU label, and fak binary.
<private-swebench-compare> \
  --engine vllm \
  --model zai-org/GLM-5.2-FP8 \
  --served-model-name glm-5.2 \
  --raw-base-url http://127.0.0.1:8000/v1 \
  --verified-count 20 \
  --skip-engine-serve \
  --require-tool-calls \
  --require-grade \
  --require-gpu-name H200 \
  --run-dir <private-swebench-run> \
  --preflight-only

# 4b. Run the 20-task SWE-bench Verified comparison. This reuses raw vLLM and
# lets the driver start a fresh fak gateway in front of it.
<private-swebench-compare> \
  --engine vllm \
  --model zai-org/GLM-5.2-FP8 \
  --served-model-name glm-5.2 \
  --raw-base-url http://127.0.0.1:8000/v1 \
  --verified-count 20 \
  --skip-engine-serve \
  --require-tool-calls \
  --require-grade \
  --require-gpu-name H200 \
  --run-dir <private-swebench-run>

Use --require-gpu-name B200 or an empty --require-gpu-name "" when the node is not an H200. The requirement is an anti-mislabel guard, not the hardware rule; the hardware rule is the preflight verdict.

Agentic Floor Series

These commands can run before or after the live vLLM run. They should be recorded as fak-native mechanism evidence, not as vLLM results:

go run ./cmd/fak swebench compare --difficulty "$FAK_SWEBENCH_DIFFICULTY" \
  --workers 1,2,4,8 --limit 20 --with-adjudication \
  --out experiments/vllm/swebench-20-fak-floor.json \
  --md experiments/vllm/swebench-20-fak-floor.md

go run ./cmd/fak turntax --suite turntax-airline \
  --out experiments/vllm/turntax-airline.json
go run ./cmd/sessionbench -synthetic smollm2-135m -turns 50 -agents 5 \
  -prefix 2048 -decode 32 -result 64 \
  -out experiments/vllm/sessionbench-synthetic.json
go run ./cmd/fanbench -profile research -trials 12 \
  -out experiments/vllm/fanbench-research.json \
  -csv experiments/vllm/fanbench-research.csv
go run ./cmd/radixbench -live=false \
  -out experiments/vllm/radixbench-synthetic.json

The SWE-bench floor uses the same 20-instance scale as the live run, but it is deterministic geometry/adjudication evidence. The live private compare runner artifact is the resolve-rate evidence.

Completion Bar

The benchmark is complete only when all of these are true:

preflight.json says the serving node is READY or READY_PENDING_INSTALL.
raw-vllm-vs-fak-gateway-contract.json records result_claim_allowed=false, the two arms (raw-vllm, fak-gateway), the shared GLM-5.2/vLLM identity, the 20-task selection, budgets/retry policy, required metrics, and result artifact paths.
The run contract’s required metrics include pass rate, safe-completion rate, prompt/completion/total token counts, tool-call counts, agent/grade/gateway latency, token and wall-clock cost proxies, policy blocks, and evidence completeness.
The run contract frames any positive result as a same-model harness gain, not GLM-5.2 model superiority or a fak-native kernel throughput win over vLLM.
full-size-serving-witness.json has summary.full_size_serving_witness == "PASS".
adjudication-tax-witness.json reports measured raw-vLLM and fak-gateway legs.
SWE-bench COMPARE-PREFLIGHT.json passes and records config plus runtime metadata for the runner, SWE-bench harness, vLLM, endpoints, and GPU guard. compare.json is for the zai-org/GLM-5.2-FP8 checkpoint served as glm-5.2 on raw vllm, with two arms, raw-vllm and fak-gateway, each with 20 submitted instances, a matching selection_instance_ids list, a passing OpenAI tool-call self-test, and official harness grades with grade_rc == 0, arm-specific run_id, submitted == 20, report_path, grade_log, and resolved_ids drawn from the selected instance IDs. The same run directory has DONE.rc equal to 0.
The fak-native floor artifacts pass their identity checks: SWE-bench floor has 20 instances and workers 1,2,4,8; turntax is turntax-airline with safety floor intact; sessionbench is synthetic smollm2-135m at T=50,C=5,P=2048; fanbench is the research grid with 12 trials; radixbench is the synthetic cache-hit/policy-eviction witness.
Any number copied into docs is linked from BENCHMARK-AUTHORITY.md with the artifact path and reproduce command. The generated runner’s final gate emits experiments/vllm/glm52-agentic-battery/BENCHMARK-AUTHORITY-DRAFT.md only after every required artifact passes.

Until then, the honest claim is only: the benchmark is wired and ready to run on a GLM-5.2-capable serving node.

GPU-server GLM-5.2 vs vLLM agentic benchmark plan

A pending-measurement plan for running GLM-5.2 on vLLM through fak's gateway across the serving-agent benchmarks and a 20-task SWE-bench Verified slice, with artifact requirements and honesty fences.