method · first seen 8 Oct 2026
softmax attention
Attention mechanism normalizing query-key scores with softmax to weight values.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It MustPapers · 8 Oct 2026 · Simon Gabet, Etienne Boursier, Claire Boyer
- Counting with Attention and LayerNorm: Norm Scaling and Representation GeometryPapers · 12 Oct 2026 · Ayyub Shaffy, Manoj Dhakal, Safal Shrestha, Keith W. Ross
- Exact Fusion of Softmax Attention Heads Can Require Exponentially Many AtomsPapers · 2 Nov 2026 · Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi, Naohiko Matsuda
- Theoretical Analysis of Muon on Transformers for Next-Token PredictionPapers · 3 Nov 2026 · Shengwei Wu, Yixuan Yang, Gao Huang, Song Li
- Softmax Attention with Any Temperature in Fixed Dimension: Matching the Best Known Maximum-Inner-Product BoundsPapers · 4 Nov 2026 · Zixuan Huang, Guozhang Chen
- LENS: Replacing LayerNorm and Softmax in Transformers with Learnable Normalization StatesPapers · 15 Nov 2026 · Yuhao Liu, Salim Ullah, Akash Kumar
Latest sources
- LENS: Replacing LayerNorm and Softmax in Transformers with Learnable Normalization StatesPapers · 15 Nov 2026 · Yuhao Liu, Salim Ullah, Akash Kumar
- Softmax Attention with Any Temperature in Fixed Dimension: Matching the Best Known Maximum-Inner-Product BoundsPapers · 4 Nov 2026 · Zixuan Huang, Guozhang Chen
- Theoretical Analysis of Muon on Transformers for Next-Token PredictionPapers · 3 Nov 2026 · Shengwei Wu, Yixuan Yang, Gao Huang, Song Li
- Exact Fusion of Softmax Attention Heads Can Require Exponentially Many AtomsPapers · 2 Nov 2026 · Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi, Naohiko Matsuda
- Counting with Attention and LayerNorm: Norm Scaling and Representation GeometryPapers · 12 Oct 2026 · Ayyub Shaffy, Manoj Dhakal, Safal Shrestha, Keith W. Ross
- Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It MustPapers · 8 Oct 2026 · Simon Gabet, Etienne Boursier, Claire Boyer