method · first seen 4 Oct 2026
sparse attention
Also: local attention, sub-quadratic attention
Attention computation restricted to a subset of query-key pairs to reduce cost.
Sparse attention restricts each query to a subset of keys, cutting the quadratic cost of full self-attention. The sources are mostly incremental arXiv papers applying it to diffusion transformers for high-resolution image and video generation: dynamic masks for joint video-audio training, closing the dense-sparse quality gap, token-to-tile GPU execution, position-content hybrid masks, and mobile NPU deployment. Other work targets long-context LLM decoding, using softmax coverage as a quality check on KV-cache selection, and graph transformers using randomized near-neighbor attention. The few GitHub repos are paper implementations and experimental frameworks. The idea itself dates to Sparse Transformers and Longformer around 2019-2020.
The concept is long established. The current activity is a steady stream of refinements, mainly to make video diffusion and long-context inference cheaper on real hardware, and does not mark a new idea breaking out.
Analyst note (established), 17 Nov 2026.
Mentions per week
PapersCodeDiscussion
Rank 145 on 13 Nov 2026 · 37 mentions in 28 days · 37 independent groups · papers → code · first seen 6 weeks ago · acceleration +0.21
Earliest sources
- Prism: Dynamic Sparse Attention for Native 2K Joint Video-Audio Generation Model TrainingPapers · 4 Oct 2026 · Shuyuan Tu, Qi Tian, Yinming Huang, Yue Wu, Xintong Han, Kaihang Pan, Weijie Kon…
- SFA: A Statistical Physics-Inspired Diffusion Transformer for Efficient and High-Quality Image GenerationPapers · 5 Oct 2026 · Zixuan Huang, Tianhao You, Guozhang Chen
- MC-Sparse: Deconstructing and Closing the Dense-Sparse Attention Gap in Diffusion TransformersPapers · 5 Oct 2026 · Jiarui Chen, Zeqiang Lai, Jiangshan Wang, Ziheng Ouyang, Ye Huang, Xiangyu Yue,…
- Backend-Agnostic Sparse Attention for Fast High-Resolution Visual GenerationPapers · 6 Oct 2026 · Liao Ma, Jiayi Song, Yunfeng Wu, Songhua Liu, Peilin Zhao
- SPIN: Shadow Predictive Indexer for Sparse AttentionPapers · 6 Oct 2026 · Yao Fu, Jiahan Chang, Yueying Li, Ritchie Zhao, Bryce Long, Mohammad Mahdi Kaman…
- sybil-solutions/qwen38-flash-next-b70-offloadCode · 6 Oct 2026 · sybil-solutions · 22 stars
Latest sources
- Sparse Attention for Visual Generation: Toward Algorithm-System Co-DesignPapers · 16 Nov 2026 · Majun Zhang, Yuan Guo, Decheng Wu, Jiayang Xu, Changhao Pan, Fan Zhuo, Xiaoda Ya…
- QASK-Edit: Adaptive Sparse Attention Across Queries and Key Sources for Efficient Video EditingPapers · 16 Nov 2026 · Hengrui Liu, Wenbo Xu, Ziqi Sheng, Wei lu
- PoCoAttn: Position-Content Collaborative Sparse Attention for Fast and High-Fidelity Video GenerationPapers · 15 Nov 2026 · Jianwen Wu, Meiling Wang, Junkang Zhou, Yipeng Shen, Haoran You
- dit.xpu : Taming DiT Sparse Inference on Mobile SoCsPapers · 14 Nov 2026 · Jiayu Chen, Jinye Hu, Maoliang Li, Anqian Cen, Hailong Zou, Zihao Zheng, Xinming…
- Token2Tile: Token-Wise Sparse Attention with Tile Execution through a Low-Rank Space for Video DiTsPapers · 13 Nov 2026 · Xuecheng Wu, Aichen Feng, Jinyan Chen, Qingjie Zhao, Julien Lai
- NearFormer: Linear-Size Sparse Attention via Randomized Near NeighborsPapers · 9 Nov 2026 · Ali Parviz, Alex Cloninger, Gal Mishne
- Coverage-Verified Sparse Attention: Closed-Loop Quality Control for Long-Context LLM DecodingPapers · 7 Nov 2026 · Yan-Cheng Li, Chun-Yi Lee
- OneReel: Efficient Multi-Scene Video Generation with Scene-Structured AttentionPapers · 7 Nov 2026 · Wei Zhang, Yuchen Wang, Tianrui Feng, Hongrui Wu, Chenfeng Xu, Fan Lai
- ADIndex: Layer-Adaptive Sparse Attention Indexing via Group-Level DistillationPapers · 7 Nov 2026 · Jin-Fei Qi, Yiduo Li, Jiang-Xin Shi, Kan Liu, Tao Lan, Hanlin Tang, Yu-Feng Li
- ESTRA: Enabling Extreme Attention Sparsity for Training-Free Video Diffusion AccelerationPapers · 7 Nov 2026 · Enhuai Liu, Yunke Wang, Yutong Wang, Changming Sun
- Attend at the Right Resolution: Head-Adaptive Sparse Attention via Query-Key Interaction StructurePapers · 6 Nov 2026 · Wajih Hassan Raza, Ahsan Bilal, Yidan Shen, Renjie Hu, Xin Fu, Yu Wen
- InstaGen: Group-Shared Importance Sampling for Fast Video GenerationPapers · 5 Nov 2026 · Kislay Aditya Oj, Kumar Krishna Agrawal, Sri Surya Varma Datla, Aditya Desai