architecture · first seen 7 Oct 2026
multi-head self-attention
Also: MHA, MHSA, multi-head attention
Attention mechanism running several attention heads in parallel over projected query, key and value representations.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Gaussian Equivalence for Multi-Head Self-AttentionPapers · 7 Oct 2026 · Tomohiro Hayase, Ryo Karakida
- Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in TransformersPapers · 9 Oct 2026 · Zhenke Duan, Xin Li, Jiqun Pan, Dong Xiaofei, Hanwen Ning, Xinyuan Song
- Choosing the Number of Attention Heads from Intrinsic Dimension: A Small-Scale StudyPapers · 14 Oct 2026 · Arjun Boya, Charan Sai Bandaru, Masthanaiah Pillimetla, Kavana Chidanandappa
- Formally Verified Autoresearch for Analyzing the Expressivity of Multi-Head AttentionPapers · 18 Oct 2026 · Karthik Viswanathan, Plamen Dimitrov, Stefano Rocca, Sarayu Sirikonda
- Head Dimension Matters in Language Models for Long-Context RetrievalPapers · 23 Oct 2026 · Seiya Tokui, Shoichiro Yamaguchi, Sosuke Kobayashi
- prabeakarve/microgptCode · 5 Nov 2026 · prabeakarve · 51 stars
Latest sources
- prabeakarve/microgptCode · 5 Nov 2026 · prabeakarve · 51 stars
- Head Dimension Matters in Language Models for Long-Context RetrievalPapers · 23 Oct 2026 · Seiya Tokui, Shoichiro Yamaguchi, Sosuke Kobayashi
- Formally Verified Autoresearch for Analyzing the Expressivity of Multi-Head AttentionPapers · 18 Oct 2026 · Karthik Viswanathan, Plamen Dimitrov, Stefano Rocca, Sarayu Sirikonda
- Choosing the Number of Attention Heads from Intrinsic Dimension: A Small-Scale StudyPapers · 14 Oct 2026 · Arjun Boya, Charan Sai Bandaru, Masthanaiah Pillimetla, Kavana Chidanandappa
- Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in TransformersPapers · 9 Oct 2026 · Zhenke Duan, Xin Li, Jiqun Pan, Dong Xiaofei, Hanwen Ning, Xinyuan Song
- Gaussian Equivalence for Multi-Head Self-AttentionPapers · 7 Oct 2026 · Tomohiro Hayase, Ryo Karakida