Is witnessed span-attention a valid reward? Verdict, aggregation, and failure modes

This is the deliverable for #861, the first prior-art survey under the reward-over-spans epic (#860). The question is narrow: before fak uses attention mass as a reward over context spans, is attention a valid importance signal at all?

Verdict:

Witnessed span-attention is defensible as a cheap, relative, span-level proposal signal. It is not defensible as a standalone causal explanation or as an accepted reward until it correlates with an ablation witness such as exact-eviction leave-one-out.

That verdict is deliberately weaker than “attention is explanation.” It is also stronger than “attention is useless.” The literature says raw attention can be noisy and manipulable, but it also retains signal under better tests, especially when used as a relative ranking and checked against perturbation, gradient, or ablation evidence.

What the debate permits

Source What it says Rule for fak
Jain and Wallace, “Attention is not Explanation” Attention weights can be poorly correlated with gradient importance, and different attention distributions can preserve similar predictions. Source: https://arxiv.org/abs/1902.10186. Do not call raw attention a causal explanation. Do not reward spans solely because a heatmap is high.
Wiegreffe and Pinter, “Attention is not not Explanation” Whether attention explains depends on the definition of explanation and the test design; adversarial attention alone does not settle usefulness. Source: https://arxiv.org/abs/1908.04626. Keep attention as a candidate signal, but define the test that promotes or refutes it.
Serrano and Smith, “Is Attention Interpretable?” Attention noisily predicts importance but is not a fail-safe indicator; gradient rankings can better predict effects in some cases. Source: https://arxiv.org/abs/1906.03731. Use attention as a prior, not the final label; keep gradient or LOO as fallback producers.
Abnar and Zuidema, “Quantifying Attention Flow in Transformers” Raw attention is less reliable because information mixes across layers; rollout and flow better approximate token relevance against ablation/gradient scores. Source: https://arxiv.org/abs/2005.00928. Avoid single-layer or last-layer heatmaps. Aggregate across layers/heads/queries, and treat rollout/flow as an offline cross-check when the goal is explanation rather than direct KV-read accounting.

The common denominator is a testable posture: attention may be useful when the claim is relative and operational, but it must be validated against a stronger witness for the model, task, and span type in question.

The aggregation fak should use

For the first reward producer, use the signal fak already witnesses:

  1. Emit post-softmax attention rows from the model seam (internal/model/attn_observer.go).
  2. Attribute each row’s key-position weights to the owning semantic span through the From/Len ledger (internal/kvmmu/attention.go).
  3. Sum across heads, layers, and consumer query positions into per-span mass.
  4. Normalize by total emitted mass and by the expected attention mass for that span’s position and length.
  5. Use the result as a relative ranking, then validate the top/bottom candidates with exact-eviction leave-one-out (internal/model/span_reward_shadow.go).

The default aggregation is therefore:

raw_mass(s) = sum over consumer rows r, heads h, layers l, positions p in span(s)
              attention(l, h, r, p)

proxy_reward(s) = success_gate * recency_discount *
                  max(0, raw_mass(s) - expected_mass(position(s), len(s)))

Important choices:

Promotion rule

Attention is accepted only after the #866-style shadow report proves it for a slice:

This keeps internal/ctxplan/snfitness.go honest: the RSI fitness path remains shadow-only until correlation against exact-eviction LOO exists.

Failure modes to guard

Decision for the reward epic

Use witnessed span-attention in the epic, but name it accurately:

The implementation order is:

  1. Keep the current post-softmax row witness and span attribution ledger.
  2. Ensure reward scoring uses consumer/probe rows, not span-construction rows.
  3. Apply #862 confound normalization before any correlation claim.
  4. Run #866 exact-eviction LOO on top/bottom candidates.
  5. Promote attention only for slices that return CORRELATE.

If this fails, #863 already defines the fallback menu: use exact LOO deltas directly, or use gradient/context-attribution producers through the same consumer seam.

Sources

Primary sources:

Local source files: