Article: A new “Attention Sink Detector” spots tokens that hoard most of the attention mass during the prefill stage of large-language-model (LLM) inference. The tool shows that removing these sink tokens from a key-value (KV) cache cripples model performance, making them essential anchors for stable generation.

Why the prefill stage matters

Most LLM research zeroes in on the token-by-token generation loop, but the work that happens before the first token—prefill—sets the tone for everything that follows. During prefill the model processes the entire prompt, builds the KV cache, and spreads attention across every position. Because the softmax that produces attention weights must sum to 1, the model cannot say “nothing is relevant.” It dumps the leftover probability mass onto a token that is easy to address, usually the first token in the sequence. That token becomes an attention sink.

What an attention sink is, in plain terms

In a Transformer, each head computes a weight for every token pair. When the distribution skews heavily, a single token may receive a disproportionate share of the attention weight. The phenomenon isn’t a bug; it follows from the softmax constraint. The first token (often a beginning-of-sentence marker) acts as a “release valve” for the residual probability that cannot be assigned elsewhere.

Stakes for KV-cache management

KV caches store the key and value vectors for every token that has been processed, enabling fast look-ups during generation. In long-context scenarios, practitioners prune the cache to stay within GPU memory limits. The new detector shows that indiscriminately deleting tokens that appear “unimportant” also deletes the very sinks the model relies on, causing performance to crash. Pinning sink tokens in the cache preserves the model’s internal equilibrium and keeps generation stable.

How the detector works

  • Eager attention: The implementation bypasses fast kernels such as FlashAttention, which collapse the full attention matrix, and instead records raw attention scores for every head.
  • Layer-wide aggregation: Scores are averaged across all layers and heads to produce a single attention-mass profile per token.
  • Log-median absolute deviation (MAD): Because attention weights are heavily right-skewed, a simple mean-standard-deviation test misclassifies normal variance as outliers. Log-transforming the weights normalises the distribution; applying MAD then flags tokens whose attention mass exceeds the typical spread.

Two classes of sinks uncovered

  1. True sinks – The beginning-of-sentence (BOS) token consistently absorbs a massive chunk of attention, regardless of the prompt content.
  2. Structural sinks – Tokens that belong to system-level prompts, such as the markers used in ChatML (<im_start>, <im_end>), form small clusters that also attract attention. They act as logical boundaries, helping the model separate user messages from system instructions.

Counter-point: Do we really need the raw matrix?

The author points out that without the raw numbers the sink phenomenon stays hidden, and any cache-pruning policy based on incomplete data risks destabilising the model.

What to watch next

The detector is the second installment in a four-part series that began with an entropy tracker for generation quality. The upcoming part will integrate sink detection with entropy-guided speculative decoding. The final study will be an empirical study.

Takeaway: Attention sinks are not an obscure quirk; they are structural pillars that keep a Transformer’s attention distribution well-behaved during prefill. Any KV-cache pruning strategy that ignores them will jeopardise model stability. Detecting and preserving these tokens is a low-cost safeguard that can make long-context inference both reliable and efficient.