Hardware bench plan — 20260903T053000Z
Auto-generated by tools/bench_plan.py --md. Do not hand-edit; re-run the tool (or the FleetBenchPlanDoc task) to refresh.
PLAN ONLY – no benchmark was run; every figure is a literal catalog fact and all ages are computed against –now.
Generated from catalog.json (last_updated 2026-09-03T05:30:00Z): 7 bench-nodes, 70 feasible cells (23 empty coverage holes, 3 with a recorded baseline), 7 infeasible cells, 95 runs on record.
Coverage matrix (bench-node × workload-kind)
. = empty feasible cell (a coverage hole) · - = infeasible on this hardware (e.g. a CUDA kind on the Apple mac) · a number = runs on record.
| machine | model | gpu | qwen36 | radix | session | fan | agent-live | turn-tax | parity | livecodebench | concept-bench |
|---|---|---|---|---|---|---|---|---|---|---|---|
| a100 | . | . | . | . | . | . | . | . | . | - | . |
| cpu-server-a | . | - | . | . | . | . | . | . | . | - | . |
| gcp-a3-high-h100-1g | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | - | 1 |
| gcp-g2-l4 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | - | 1 |
| gcp-g2-l4-32 | 2 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | - | 1 |
| node-macos-a | 3 | - | 5 | 1 | 1 | . | 1 | . | 1 | . | . |
| workstation-a | 5 | 1 | 9 | 2 | 1 | 2 | 5 | 2 | 4 | 2 | 1 |
Next test per machine
| machine | next workload | model / precision | intent | score | why |
|---|---|---|---|---|---|
| a100 | gpu-benchmark | qwen2.5-3b / Q8_0 | learn-collect | 0.8356 | a100 has 0 runs (last_run=None); the gpu-benchmark cell has never run – a first-ever measurement on this node. |
| cpu-server-a | model-benchmark | SmolLM2-135M-Instruct / q8_0 | coverage | 0.7326 | cpu-server-a has run other workloads but the model-benchmark cell is empty – a coverage gap to fill (SmolLM2-135M-Instruct). |
| gcp-a3-high-h100-1g | model-benchmark | SmolLM2-135M-Instruct / q8_0 | benchmark | 0.3607 | gcp-a3-high-h100-1g/model-benchmark ran 1x, last 50.24d ago (re-check 7d) but recorded no number – re-run to capture one. |
| gcp-g2-l4 | model-benchmark | SmolLM2-135M-Instruct / q8_0 | benchmark | 0.3583 | gcp-g2-l4/model-benchmark ran 1x, last 50.24d ago (re-check 7d) but recorded no number – re-run to capture one. |
| gcp-g2-l4-32 | qwen36 | qwen3.6-27b / q8 | benchmark | 0.3594 | gcp-g2-l4-32/qwen36 ran 1x, last 50.19d ago (re-check 14d) but recorded no number – re-run to capture one. |
| node-macos-a | livecodebench | qwen3.6-27b / official | coverage | 0.6287 | node-macos-a has run other workloads but the livecodebench cell is empty – a coverage gap to fill (qwen3.6-27b). |
| workstation-a | model-benchmark | SmolLM2-135M / q8_0 | regression | 0.4307 | workstation-a/model-benchmark has a recorded 31.02 tok/s baseline aged 50.2d (re-check every 7d) – re-measure to catch drift. |
Do next — global ranked plan
| # | machine | workload | intent | score | suggested command (hint) |
|---|---|---|---|---|---|
| 1 | a100 | gpu-benchmark | learn-collect | 0.8356 | go run -tags cuda ./cmd/gpucheck |
| 2 | a100 | model-benchmark | learn-collect | 0.8326 | go run ./cmd/modelbench -quant |
| 3 | a100 | radix-benchmark | learn-collect | 0.8326 | go run ./cmd/radixbench |
| 4 | a100 | qwen36 | learn-collect | 0.812 | fak serve + fak agent (qwen3.6-27b via gateway) |
| 5 | a100 | agent-live | learn-collect | 0.77 | go run ./cmd/fak agent --task <task> |
| 6 | a100 | concept-benchmark | learn-collect | 0.77 | go run ./cmd/conceptbench --replay cmd/conceptbench/testdata/replay |
| 7 | a100 | fan-benchmark | learn-collect | 0.77 | go run ./cmd/fanbench |
| 8 | a100 | parity | learn-collect | 0.77 | go run ./cmd/paritybench |
| 9 | a100 | session-benchmark | learn-collect | 0.77 | go run ./cmd/sessionbench |
| 10 | a100 | turn-tax | learn-collect | 0.77 | go run ./cmd/fak turntax --suite turntax-airline |
By intent
The planner guarantees all four operator intents are surfaced.
Benchmark perf
- workstation-a → gpu-benchmark (score 0.384): workstation-a/gpu-benchmark ran 1x, last 76.22d ago (re-check 7d) but recorded no number – re-run to capture one.
- workstation-a → session-benchmark (score 0.384): workstation-a/session-benchmark ran 1x, last 50.21d ago (re-check 14d) but recorded no number – re-run to capture one.
- node-macos-a → session-benchmark (score 0.3607): node-macos-a/session-benchmark ran 1x, last 75.67d ago (re-check 14d) but recorded no number – re-run to capture one.
- node-macos-a → radix-benchmark (score 0.3607): node-macos-a/radix-benchmark ran 1x, last 71.82d ago (re-check 14d) but recorded no number – re-run to capture one.
- node-macos-a → agent-live (score 0.3607): node-macos-a/agent-live ran 1x, last 70.02d ago (re-check 30d) but recorded no number – re-run to capture one.
Learn / collect new data
- a100 → gpu-benchmark (score 0.8356): a100 has 0 runs (last_run=None); the gpu-benchmark cell has never run – a first-ever measurement on this node.
- a100 → model-benchmark (score 0.8326): a100 has 0 runs (last_run=None); the model-benchmark cell has never run – a first-ever measurement on this node.
- a100 → radix-benchmark (score 0.8326): a100 has 0 runs (last_run=None); the radix-benchmark cell has never run – a first-ever measurement on this node.
- a100 → qwen36 (score 0.812): a100 has 0 runs (last_run=None); the qwen36 cell has never run – a first-ever measurement on this node.
- a100 → agent-live (score 0.77): a100 has 0 runs (last_run=None); the agent-live cell has never run – a first-ever measurement on this node.
Prevent regression
- workstation-a → model-benchmark (score 0.4307): workstation-a/model-benchmark has a recorded 31.02 tok/s baseline aged 50.2d (re-check every 7d) – re-measure to catch drift.
- node-macos-a → parity (score 0.2008): node-macos-a/parity has a recorded 7.61 tok/s baseline aged 0.02d (re-check every 30d) – re-measure to catch drift.
- node-macos-a → model-benchmark (score 0.1164): node-macos-a/model-benchmark has a recorded 38.07 tok/s baseline aged 0.02d (re-check every 7d) – re-measure to catch drift.
Fill coverage gaps
- cpu-server-a → model-benchmark (score 0.7326): cpu-server-a has run other workloads but the model-benchmark cell is empty – a coverage gap to fill (SmolLM2-135M-Instruct).
- cpu-server-a → radix-benchmark (score 0.7326): cpu-server-a has run other workloads but the radix-benchmark cell is empty – a coverage gap to fill (SmolLM2-135M).
- cpu-server-a → qwen36 (score 0.712): cpu-server-a has run other workloads but the qwen36 cell is empty – a coverage gap to fill (qwen3.6-27b).
- cpu-server-a → agent-live (score 0.67): cpu-server-a has run other workloads but the agent-live cell is empty – a coverage gap to fill (agent workload).
- cpu-server-a → concept-benchmark (score 0.67): cpu-server-a has run other workloads but the concept-benchmark cell is empty – a coverage gap to fill (agent workload).
Frontier & honesty call-outs
a100is a total coverage frontier — every feasible workload-kind (10) has never run here.cpu-server-ais a total coverage frontier — every feasible workload-kind (9) has never run here.
Excluded (not a bench target)
desktop— role=agent-host (run-on-bench-nodes-by-default).
Methodology & honesty
Each feasible (machine × workload-kind) cell is scored on five dimensions, each in [0,1]:
| dimension | weight | intent | reads |
|---|---|---|---|
| coverage_gap | 0.34 | coverage | run count in the cell (empty = 1.0) |
| machine_novelty | 0.20 | learn-collect | machine total runs / last_run |
| staleness_overdue | 0.16 | regression | newest cell timestamp vs –now, per-kind interval |
| baseline_drift | 0.16 | regression | a recorded peak_tok_per_sec aging past its interval |
| model_diversity | 0.14 | coverage | anti-monoculture: how rare the proposed model is |
Cells sort empty-tier-first (coverage dominance), then by weighted score. Feasibility (no CUDA on the mac) is applied before scoring. --now is injected, never read from the wall clock, so a fixed stamp yields identical output. marginal-information-gain (novelty / diversity) is a heuristic proxy, not a measured quantity — the ranking is a guide, not a verdict. This tool only PLANS; a run is a later action on the remote bench-node.