method · first seen 11 Oct 2026
continual pretraining
Also: CPT, incremental training, continual pre-training, continued pre-training
Further pretraining an existing model on new data distributions to add capabilities such as new languages.
Mentions per week
PapersCodeDiscussion
Rank 190 on 1 Nov 2026 · 9 mentions in 28 days · 9 independent groups · papers · first seen 3 weeks ago · acceleration -0.83
Earliest sources
- In-Place Indic Tokenizer Expansion and Continual Pretraining of Nemotron-3-Nano-30B-A3BPapers · 11 Oct 2026 · Avinash Benki, Prasanjith Pasunuti, Sutanshu Saksena Raj, Parijat Chatterjee, Na…
- Latent Taxonomies: How Property Induction Develops in Language Model Pre-trainingPapers · 11 Oct 2026 · Jan Sobotka, Ekaterina Goliakova, Marcin Detyniecki, Antoine Bosselut, Xavier Re…
- Continual pretraining Prostate MRI ViT Encoder for task alignmentPapers · 14 Oct 2026 · Bilal A. Sidiqi, Yipei Wang, Shaheer U. Saeed, Shiqi Huang, Daniel C. Alexander,…
- Staged Continual Adaptation of Multimodal Foundation Models for Japanese Financial DocumentsPapers · 20 Oct 2026 · Genshin Kakimoto, Atsushi Yanagisawa
- Scaling in Games: Continued Pre-Training For Embodied Agents In Diverse Virtual WorldsPapers · 20 Oct 2026 · Kuan Zhang, Yukun Chen, Zhihao Yang, Yue Su, Xiangnan Wu, Zirong Chen, Run Luo,…
- Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling LawPapers · 21 Oct 2026 · Ziqing Qiao, Xiaoyue Xu, Sikui Zhang, Yinuo Xu, Bingxiang He, Chaojun Xiao, Xu H…
Latest sources
- Augmenting Pretrained Language Models with N-gram Embeddings through Incremental TrainingPapers · 16 Nov 2026 · Shaohua Hu, Xusheng Gu, Haofei Yang, Minghao Shao, Nanhui Zhao, Caoyun Fan, Fang…
- Beyond Passive Imitation: On-Policy Mid-Training Distillation for Language ModelsPapers · 10 Nov 2026 · Ruize Zheng, Xuyang Liu, Siyu Ren, Xiaobing Tu, Yinggui Wang, Jinkui Ren, Xianta…
- Jais 3: Introducing Arabic-centric Agentic AIPapers · 6 Nov 2026 · Team IFM, Haonan Li, Hadi Abdine, Nikhil Ranjan, Mohamed Anwar, Ahmad Chamma, Am…
- SLA: Scaling Spatial-Language-Action Model via Continual Pre-trainingPapers · 5 Nov 2026 · Fuhao Li, Wenxuan Song, Qizhou Huang, Ziyang Zhou, Jingbo Wang, Junwu E, Pengwei…
- InnerLogic-Surg: An Open Surgical Foundation Model via Continual PretrainingPapers · 5 Nov 2026 · Kun Yuan, Hao Ding, Yiqing Shen, Gasper Podobnik, Benjamin Killeen, Tito Porras,…
- AssameseLM-E2B: An Open-Weight Assamese Language Model for Efficient Consumer-Grade DeploymentPapers · 4 Nov 2026 · Pragyan Phukan
- Nested Sparsity for Scalable Pretraining and Continual LearningPapers · 22 Oct 2026 · Tianzhe Chu, Druv Pai, Yi Ma
- Parameter-Efficient Adaptation of Large Language Models to a Low-Resource Agglutinative Language: A Corpus, Datasets, and a Reproducible Benchmark for the Bashkir LanguagePapers · 22 Oct 2026 · Mullosharaf K. Arabov, Svetlana S. Khaybullina
- AQL-Lite: Enabling Efficient Knowledge Acquisition for Arabic LLMs via Data-Centric Continual PretrainingPapers · 21 Oct 2026 · Houssam Eddine Boukhalfa, Mourad Bouache, Mourad Dahmane
- Don't Stop Long-Context Continual Pre-Training Too Early: An Empirical Scaling LawPapers · 21 Oct 2026 · Ziqing Qiao, Xiaoyue Xu, Sikui Zhang, Yinuo Xu, Bingxiang He, Chaojun Xiao, Xu H…
- Scaling in Games: Continued Pre-Training For Embodied Agents In Diverse Virtual WorldsPapers · 20 Oct 2026 · Kuan Zhang, Yukun Chen, Zhihao Yang, Yue Su, Xiangnan Wu, Zirong Chen, Run Luo,…
- Staged Continual Adaptation of Multimodal Foundation Models for Japanese Financial DocumentsPapers · 20 Oct 2026 · Genshin Kakimoto, Atsushi Yanagisawa