Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To test whether modern architectural variants genuinely mitigate attention sinks and positional retrieval failures at extreme context lengths, the authors evaluate mechanisms like gated and delta attention at one-million-token scale using a diagnostic suite called SinkProbe. Resolving these pathologies is critical to ensuring effective token utilization rather than degenerate first-token bias across ultra-long context windows. The investigation reveals that gated attention fails to replicate published sink-reduction benefits at this scale, showing that attention sinks are an artifact of the pretraining objective rather than the attention architecture.
Gated attention's promise of eliminating attention sinks evaporates at million-token scale, proving that sink tokens and massive activations are fundamentally driven by the training objective rather than the attention architecture.
Long context language models now advertise windows of one million tokens, but two habits limit how much of that window is used. Attention heads with nothing useful to read still spend their budget on the first token, which is called the attention sink, and where a fact sits in the context changes whether the model finds it. Gated attention cut first token attention from 46.7 percent to 4.8 percent at NeurIPS 2025, and Kimi K3 pairs that idea with Kimi Delta Attention and Attention Residuals behind a one million token window, eight times past the range where these diagnostics have been reported. This paper asks whether the fix survives that jump. We build SinkProbe, a suite that measures sink mass, massive activation, position resolved recall and the recency gap, and apply it to four small models that differ only in how they mix tokens and depth. Three results follow. The training objective produces the sink, not the architecture. Gating did not reproduce its published effect at our scale. Sink mass, activations and position bias moved independently. Code, data and the measurement protocol are released at https://github.com/sararizwan7/Attention-Mechanisms-in-1M-Context-Window