phenomenon · first seen 4 Oct 2026
data repetition in pretraining
Also: multi-epoch pretraining
Effects on language model loss and scaling when training data is seen multiple times during pretraining.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Selecting Repetition Counts Across Model Scales in Data-Constrained PretrainingPapers · 4 Oct 2026 · Ziyue WANG, T. Kanamori
- What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch PretrainingPapers · 4 Oct 2026 · Yekun Chai, Haoyi Xiong
Latest sources
- What Is a Repeated Token Worth? The Scaling Geometry of Multi-Epoch PretrainingPapers · 4 Oct 2026 · Yekun Chai, Haoyi Xiong
- Selecting Repetition Counts Across Model Scales in Data-Constrained PretrainingPapers · 4 Oct 2026 · Ziyue WANG, T. Kanamori