metrics-service study — aligned evidence and operator observability

Observed: 2026-08-29
Tracker: #10287
Receipt: study_7079004a7801dd62685c9f5ff323f40650d4cf2c52daf83a96d2c76103501f93
Source: anthony-chaudhary/metrics-service@df58aad21de603992fc7b4900ac814610092c6c3 (v0.21.16)
Disposition: study and clean-room adaptation only; no project license grant was found at the pinned revision.

Verdict

metrics-service is a coherent Go observability runtime whose useful spine is:

JSON configuration → backend adapters → deadline-aligned collection → normalized and validated snapshot → concurrent sink fan-out → bounded query and operator surfaces.

The study retained three independent follow-ons for FAK:

  1. Aligned fleet snapshots with typed partial outcomes — #10312.
  2. Typed component readiness with copyable recovery actions — #10314.
  3. Registry-driven dashboard and alert drift auditing — #10313.

Other mechanisms were marked present, watch-only, or excluded rather than filing a broad adoption epic. In particular, direct Grafana push is excluded as a default because it expands credential and external-write authority, and raw source code cannot be copied because the repository carries no license grant.

Source identity and evidence quality

The test corpus spans unit, property, fuzz, contract, smoke, and pipeline E2E styles. High-value witnesses include cmd/metrics-service/pipeline_e2e_test.go:25@df58aad…, internal/api/handler_diagnostics_contract_test.go:32,100@df58aad…, and the directory-tail cases at internal/backend/dirtail/dirtail_test.go:133-281@df58aad….

Architecture studied

Configuration and composition

internal/config/config.go:250-335@df58aad… constructs defaults and overlays JSON, while internal/config/config.go:377-430@df58aad… separates strict errors from warnings. cmd/metrics-service/main.go:20-46@df58aad… is an auditable composition root for configuration, process cancellation, pipeline, run manager, observer, Grafana, HTTP serving, and shutdown.

The mechanism is useful, but the loader does not visibly use unknown-field rejection. FAK should retain its stronger validation posture rather than copy that weakness.

Collection, partial results, and publication

internal/collector/alignment.go:20-67@df58aad… gives matched pull targets one collection window and launches them concurrently. internal/collector/collector_test.go:142-199@df58aad… distinguishes partial backend failure from total failure. internal/collector/collector.go:339-370@df58aad… validates once before concurrent sink publication, waits for sink attempts, then removes rich raw Prometheus families before long-lived storage.

This yields three transferable principles:

Storage and lifecycle

internal/sink/memory/store.go:15-82@df58aad… is a fixed-capacity chronological ring. The run lifetime is hierarchical: process context, run-duration context, then per-scrape timeout at internal/run/run.go:113,223@df58aad…. cmd/metrics-service/server.go:188-216@df58aad… bounds HTTP shutdown and closes sinks afterward.

The ring is cheap and predictable, but it relies on an immutability-after-publication rule because snapshots are stored by pointer. FAK should not generalize that into a single cross-kernel event store without preserving each receipt domain’s existing ownership, provenance, and retention contract.

Operator surface

internal/ui/page_status.go:20-80,181-248,279-291@df58aad… aggregates uptime, sinks, backend state, schemas, mappings, runs, diagnostics, and compact error details. internal/ui/ui.go:119-155@df58aad… attaches diagnostics and masks secrets. internal/ui/sse.go:12-90@df58aad… emits a deliberately bounded live payload only when the snapshot timestamp changes.

The product worldview is strong: native status should answer what is working, what is degraded, what workload is affected, and what detail remains available without requiring Grafana. Its main evidence gap is visual: no _test.go files exist under internal/ui/, so FAK’s adaptation requires a captured render witness rather than inheriting that gap.

Metrics, dashboards, and drift

internal/sink/prometheus/prometheus.go:20-176@df58aad… owns stable canonical metrics while internal/sink/prometheus/raw.go:11-196@df58aad… optionally preserves backend-specific families. internal/grafana/generate.go:20-48,206-460@df58aad… derives dashboards from schema; internal/grafana/audit.go:15-89@df58aad… finds missing coverage and deprecated references; internal/grafana/generate_test.go:93-224@df58aad… exercises real-schema generation and real-dashboard audit.

The canonical/raw split helps integration velocity, but the raw path lacks an obvious cardinality budget, family allowlist, or label-value cap. FAK should default to canonical metrics and keep any forensic path bounded, scrubbed, provenance-typed, and explicitly enabled.

Candidate decisions

1. Aligned fleet observations with typed partial outcomes — PARTIAL / DEFAULT

For: operators and benchmark authors comparing a fleet at one logical instant.
Problem: serial probes mix time windows and a single aggregate error obscures successful peers.
Today: internal/metrics/device_spine_scrape.go:122-175@7ce62aaf2 scrapes peers serially; internal/metrics/device_spine_federate.go:55-72@7ce62aaf2 federates available samples without an explicit completeness contract.
Better because: one bounded window makes evidence comparable and typed outcomes preserve independent successes.
Witness: tests for shared deadline/timestamp, mixed success, total failure, cancellation, concurrency bound, and stable ordering.

2. Component readiness with recovery actions — PARTIAL / DEFAULT

For: operators starting or diagnosing fak without an external dashboard.
Problem: process-alive and server-ready do not express which enforcement, receipt, provider, engine, cache, or optional integration component failed.
Today: internal/serverproduct/contract.go:97-235,311-345@7ce62aaf2 and internal/serverproduct/encoding.go:16-92@7ce62aaf2 provide a strong authored-spec and ready-receipt contract, but not a general degraded-state matrix.
Better because: critical failure is distinguishable from optional degradation, with exact next actions.
Witness: contract tests plus a captured operator render showing scrubbed states, criticality, last success, retry behavior, and recovery commands.

3. Metric registry and dashboard/alert drift audit — PARTIAL / DEFAULT

For: maintainers and operators evolving metrics and visualization together.
Problem: dashboards can reference retired metrics, omit required families, violate cardinality contracts, misuse counters, or lose recovery runbooks.
Today: internal/metrics/openmetrics.go:13-344@7ce62aaf2 validates and renders OpenMetrics, and internal/metrics/parity_dashboard.go:198-370@7ce62aaf2 builds one native dashboard, but there is no general registry-to-dashboard/alert audit.
Better because: the operational contract becomes deterministic and release-gated while native operation remains independent from Grafana.
Witness: negative fixtures for each finding class and a real bundled-asset pass.

Excluded or watch-only

Provenance and licensing gate

A tracked-tree search found no LICENSE, LICENCE, NOTICE, COPYING, or COPYRIGHT, and GitHub reported no license metadata. Direct dependencies appear to carry upstream licenses in the local module cache, but that does not grant a license to this repository’s source.

Therefore:

Completeness audit

Opened or inventoried:

Intentionally not treated as current implementation proof:

No submodules or vendor/ tree were present. Swagger JSON/YAML are generated release artifacts per .claude/skills/release/skill.md:44-54@df58aad…; they lack a visible generated header. The monitor registry command fak dev study-monitor --due-days 14 could not complete because a pre-existing row uses unsupported source class paper_source; this study does not alter that unrelated schema debt.

Durable outputs