method · first seen 8 Oct 2026
weight decay
Also: AdamW weight decay, weight norm control, decoupled weight decay
Regularization that penalizes large parameter values by shrinking weights during training.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network WeightsPapers · 8 Oct 2026 · Dmitrii Andriianov, Andrey Veprikov, Aleksandr Beznosikov
- MuonGS: Pretraining LLMs with Relaxed Weight-Norm ControlPapers · 27 Oct 2026 · Athanasios Glentis, Dawei Li, Mingyi Hong
- Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model SizePapers · 2 Nov 2026 · Esther Xin
- Near-Zero Token Embeddings Silently Stall Training in Gated Sequence ModelsPapers · 3 Nov 2026 · Xin Liu, Boyuan Sun, Yuxin Wang, Haoran Song, Mingjian Zhao, Mengyu Liu
- Delayed Effects of Temporary Weight Decay in Muon PretrainingPapers · 13 Nov 2026 · Md Rifat Arefin, Ravid Shwartz-Ziv, Ernie Chang, Chinnadhurai Sankar, Rylan Conw…
Latest sources
- Delayed Effects of Temporary Weight Decay in Muon PretrainingPapers · 13 Nov 2026 · Md Rifat Arefin, Ravid Shwartz-Ziv, Ernie Chang, Chinnadhurai Sankar, Rylan Conw…
- Near-Zero Token Embeddings Silently Stall Training in Gated Sequence ModelsPapers · 3 Nov 2026 · Xin Liu, Boyuan Sun, Yuxin Wang, Haoran Song, Mingjian Zhao, Mengyu Liu
- Tune the Baseline First: Most of Muon's Scaled Weight Decay Gain Is a Tuning Gap, but Its Small Real Advantage Grows With Model SizePapers · 2 Nov 2026 · Esther Xin
- MuonGS: Pretraining LLMs with Relaxed Weight-Norm ControlPapers · 27 Oct 2026 · Athanasios Glentis, Dawei Li, Mingyi Hong
- Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network WeightsPapers · 8 Oct 2026 · Dmitrii Andriianov, Andrey Veprikov, Aleksandr Beznosikov