Early-Signal Detector

method · first seen 4 Oct 2026

sparse attention

Also: local attention, sub-quadratic attention

Attention computation restricted to a subset of query-key pairs to reduce cost.

Sparse attention restricts each query to a subset of keys, cutting the quadratic cost of full self-attention. The sources are mostly incremental arXiv papers applying it to diffusion transformers for high-resolution image and video generation: dynamic masks for joint video-audio training, closing the dense-sparse quality gap, token-to-tile GPU execution, position-content hybrid masks, and mobile NPU deployment. Other work targets long-context LLM decoding, using softmax coverage as a quality check on KV-cache selection, and graph transformers using randomized near-neighbor attention. The few GitHub repos are paper implementations and experimental frameworks. The idea itself dates to Sparse Transformers and Longformer around 2019-2020.

The concept is long established. The current activity is a steady stream of refinements, mainly to make video diffusion and long-context inference cheaper on real hardware, and does not mark a new idea breaking out.

Analyst note (established), 17 Nov 2026.

Mentions per week

01419 May11 Aug10 NovPapersCodeDiscussion

PapersCodeDiscussion

Rank 145 on 13 Nov 2026 · 37 mentions in 28 days · 37 independent groups · papers → code · first seen 6 weeks ago · acceleration +0.21

Earliest sources

Latest sources

Mentioned together with

KV cachediffusion transformerspeculative decodingFlashAttentionvideo diffusion modelKV cache compressionvideo diffusion transformerdiffusion language modellinear attentionautoregressive video diffusion