phenomenon · first seen 4 Oct 2026
attention sink
Transformer attention heads concentrating large attention mass on a few early or uninformative tokens.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Underscoring the Problem: Why Softpick Fails at InitializationPapers · 4 Oct 2026 · Aryan Sood, Jaikaran Singh, Ishaan Bansal
- Spectral Geometry of Attention: From Information Routing to UncertaintyPapers · 5 Oct 2026 · Giulio Viganò, Simone Melzi, Maks Ovsjanikov
- SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language DecodersPapers · 8 Oct 2026 · Jeonghyo Song, YoungJoon Yoo
- Exploring Attention Sinks in Text-to-Image Transformers from Diffusion and Autoregressive ModelsPapers · 9 Oct 2026 · Binglei Li, Mengping Yang, Zhiyu Tan, Wenxu Qian, Junping Zhang, Hao Li
- Attention Sinks in VLMs Are Borrowed, Not SeenPapers · 11 Oct 2026 · Dengjia Zhang, Alexander Martin, Pengyu Yan, Xihan Sun, Jiaqi Li, Benjamin Van D…
- Before Attention Sinks Form: Hidden-State Geometry Reveals and Modulates Their FormationPapers · 12 Oct 2026 · Fan Yang, Yue Song
Latest sources
- Attention Sinks as Spectral Spikes: A Mechanism Analysis of Gated AttentionPapers · 16 Nov 2026 · Seojin Kim, Yehjin Shin, Noseong Park
- First-Token Attraction in Mamba DynamicsPapers · 5 Nov 2026 · Trinh T. Nguyen, Minh-Khoi Nguyen-Nhat, Duy-Tung Pham, Hoang-Son Do, Tan Minh Ng…
- Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix TokensPapers · 5 Nov 2026 · Seongtae Hong, Youngjoon Jang, Jungseob Lee, Chanjun Park, Heuiseok Lim
- Written Once, Read Differently How Language Models Own Answers Steer Their Next OnesPapers · 2 Nov 2026 · Marcus Armstrong
- Disen-Forcing: Disentangling Semantic Anchoring from Motion for Autoregressive Video DiffusionPapers · 2 Nov 2026 · Zhengyang Yu, Akio Hayakawa, Masato Ishii, Qingtao Yu, Xuesong Li, Takashi Shibu…
- What LLMs Do in the Shadows: Reading Context Reliance from the Unembedding Dark SpacePapers · 16 Oct 2026 · Alexandra Kuleshova, Andrey Volodichev, Daria Kotova, Alexey Zaytsev
- StreamErase: Towards Ultra-Fast Streaming Video ErasingPapers · 14 Oct 2026 · Zhuobai Dong, Xiaobing Tu, Xiantao Zhang, Jinkui Ren, Yinggui Wang, Yue Ma, Linf…
- WinARDet: Compressed Window Auto-Regression for Long-Horizon Vision-Language GroundingPapers · 14 Oct 2026 · Chenxu Wang, Hangyi Kuang, Bo-Wen Yin, Yuxuan Li, Quan-Sheng Zeng, Yang Zhang, X…
- Why don't Graph Transformers attend to the same token?Papers · 14 Oct 2026 · Alexandre Bismuth
- Before Attention Sinks Form: Hidden-State Geometry Reveals and Modulates Their FormationPapers · 12 Oct 2026 · Fan Yang, Yue Song
- Attention Sinks in VLMs Are Borrowed, Not SeenPapers · 11 Oct 2026 · Dengjia Zhang, Alexander Martin, Pengyu Yan, Xihan Sun, Jiaqi Li, Benjamin Van D…
- Exploring Attention Sinks in Text-to-Image Transformers from Diffusion and Autoregressive ModelsPapers · 9 Oct 2026 · Binglei Li, Mengping Yang, Zhiyu Tan, Wenxu Qian, Junping Zhang, Hao Li