method · first seen 8 Oct 2026
AdamW
Also: AdamW optimizer, decoupled weight decay Adam, Adam with decoupled weight decay
An adaptive gradient optimizer with decoupled weight decay widely used for training neural networks.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State QuantizationPapers · 8 Oct 2026 · Hanyang Li, Shao Tang, Daniel Thomas Braithwaite, Gregory Dexter, Leonardo Neves…
- Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer TrainingPapers · 12 Oct 2026 · Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi, Ganesh Talluri, Atsushi Nita…
- Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer ReparameterizationPapers · 13 Oct 2026 · Bum Jun Kim
- CAST: History-Adaptive Spectral Clipping for LLM PretrainingPapers · 14 Oct 2026 · Jiacheng Li, Jianchao Tan, Kefeng Zhang, Yuchen Xie, Xunliang Cai
- When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget BenchmarkPapers · 15 Oct 2026 · Mushir Akhtar, Hazem N. Nounou, Majdi Mansouri, Mohamed N. Nounou
- The Ephemeral Cost of Switching Optimizers Across LLM Training StagesPapers · 15 Oct 2026 · Xinyu Zhao, Davis Wertheimer, Ashutosh Sharma, Rameswar Panda, Paul Pu Liang
Latest sources
- On the Convergence of Continuous-Time AdamWPapers · 16 Nov 2026 · Bangjie Wang, Ning Ning
- Near-Optimal Methods for Generalized Lipschitz Convex OptimizationPapers · 15 Nov 2026 · Dmitry Kovalev, Grigory Malinovsky
- A Stability Analysis of AdamW: Unstable Equilibria and Non-ConvergencePapers · 8 Nov 2026 · Baiyu Su, Lizhang Chen, Qiang Liu
- Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning GapPapers · 2 Nov 2026 · Catherine Li, Jingchu Gai, Andrej Risteski, Jacob Mitchell Springer, Aditi Raghu…
- The Ephemeral Cost of Switching Optimizers Across LLM Training StagesPapers · 15 Oct 2026 · Xinyu Zhao, Davis Wertheimer, Ashutosh Sharma, Rameswar Panda, Paul Pu Liang
- When Does Muon Actually Win? A Paired Data-, Time-, and Tuning-Budget BenchmarkPapers · 15 Oct 2026 · Mushir Akhtar, Hazem N. Nounou, Majdi Mansouri, Mohamed N. Nounou
- CAST: History-Adaptive Spectral Clipping for LLM PretrainingPapers · 14 Oct 2026 · Jiacheng Li, Jianchao Tan, Kefeng Zhang, Yuchen Xie, Xunliang Cai
- Same Function, Different Trajectory: No Standard Optimizer Is Equivariant to Transformer ReparameterizationPapers · 13 Oct 2026 · Bum Jun Kim
- Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer TrainingPapers · 12 Oct 2026 · Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi, Ganesh Talluri, Atsushi Nita…
- Rounding in Preconditioner Space: Redesigning 4-bit AdamW Optimizer-State QuantizationPapers · 8 Oct 2026 · Hanyang Li, Shao Tang, Daniel Thomas Braithwaite, Gregory Dexter, Leonardo Neves…