method · first seen 4 Oct 2026
knowledge distillation
Also: KD, distillation, KL distillation, LLM distillation, logit distillation, model distillation, online distillation, teacher distillation
Training a smaller student model to imitate a larger teacher model's outputs or intermediate reasoning.
Knowledge distillation trains a smaller student model to match a larger teacher's soft outputs, logits, features or reasoning traces. The sources span many routine applications: a sparse on-policy distillation method for LLM reasoning traces, KD from imperfect teachers in low-resource audio, joint pruning and distillation for audio deepfake detection, shadow removal, a study of whether matrix-structured optimizers help KL distillation of Qwen students, and a repo for black-box LLM distillation via teacher API, rejection sampling, SFT and GRPO. The volume is almost entirely arXiv papers that use distillation as one component among others. The weekly counts are already falling, which fits steady background usage rather than a new idea gaining ground.
Distillation has been a standard compression and transfer technique since 2015. The narrower variants seen here, such as on-policy and black-box LLM distillation, are the parts worth tracking, but the umbrella concept is not new.
Analyst note (established), 17 Nov 2026.
Mentions per week
PapersCodeDiscussion
Rank 38 on 14 Nov 2026 · 178 mentions in 28 days · 174 independent groups · papers → discussion → code · first seen 6 weeks ago · acceleration -0.40
Earliest sources
- ResOPD: Tail Residualization for Sparse On-Policy DistillationPapers · 4 Oct 2026 · Penghui Yang, Long Xing, Xuanlang Dai, Ziyu Liu, Kai Chen, Yuhang Zang
- Learning from imperfect teachers for low-resource acoustic generalizationPapers · 4 Oct 2026 · Shuanglin Li, Ruxiao Qian, Jian Liu, Haijun Lin, Wenwu Wang, Siyang Song
- Task-Aware Joint Pruning and Distillation for Efficient Audio Deepfake DetectionPapers · 4 Oct 2026 · Miao He, Peng Cheng, Zhongjie Ba, Qing Wen, Li Lu, Xin Yang, Kui Ren
- Shadow Feature Refinement Network: Progressive Feature Refinement based on Knowledge Distillation for Effective Shadow RemovalPapers · 4 Oct 2026 · Donghyun Han, Byoung-Dai Lee
- Distributed Subliminal Learning: Replacing Model Updates with Random-Carrier OutputsPapers · 4 Oct 2026 · Dario Fenoglio, Gabriele Dominici, Martin Gjoreski, Marc Langheinrich
- Investigating Model Compression for Neural Machine Translation in the Biomedical DomainPapers · 4 Oct 2026 · Maria Zafar, Souhail Bakkali, Rejwanul Haque
Latest sources
- Déjà-Events: Multi-view 3D Geometry with Event Cameras via Sparsity-Aware Adaptive GatingPapers · 16 Nov 2026 · Arda Cinar Demirtas, Pablo Varela Bernal, Marc Pollefeys, Sunghwan Hong
- Towards wearable-scale, transition-aware sleep staging for clinical populationsPapers · 16 Nov 2026 · Suvadeep Maiti, Keya Sinha, Raju S. Bapi
- Stochastic Self-Distillation for Flow-Based Vision-Language-Action ModelsPapers · 16 Nov 2026 · Jingru Guo, Alessandra Russo, Joey Bose, Christos Ziakas
- Trading Parameters for Accuracy in MambaVision for Chest Radiograph ClassificationPapers · 15 Nov 2026 · Qaiser Farooq, Tehseen Zia
- SGIP: Learning Scene-Grounded Interaction Priors for Scene-Aware Human Motion PredictionPapers · 15 Nov 2026 · Donghyeon Soon, Jiil Yoon, Hyeondong Kim, Daehee Park
- PanDWM: Panoramic 4D Driving World ModelsPapers · 15 Nov 2026 · Jiajun Sun, Zeyu Zhang, Zeyu Ren, Guangfeng Jiang, Yang Zhao, Hao Tang
- Duoasa/Duoasa-skillCode · 14 Nov 2026 · Duoasa · 150 stars
- StyleJudge: A Human-Validated, In-Context-Learning Vision-Language Evaluator for Artistic Style TransferPapers · 14 Nov 2026 · Lokesh, Pramukto Mandal, Somanshu Juneja, Jashnoor Singh, Amit Kumar, Pranav Adl…
- Do Pretraining Optimizer Gains Transfer to KL Distillation?Papers · 14 Nov 2026 · Lavinia Ghita, Dhruv Desai
- Staged Semantic Distillation: Preserving Free-Form Semantics in a Low-Compute Non-LLM Surrogate of a Staged LLM AgentPapers · 14 Nov 2026 · Zhiyang Niu, Huayuan Shen, Chuanjie Li, Mengmeng Zhao, Jiajia Zhang
- PROOF: Perceptive Robustness via Out-of-Distribution Fallback for Legged LocomotionPapers · 13 Nov 2026 · Siyuan Wang, Dongjie Zhu, Jinxuan Lu, Wenjun Xu, Jiankun Wang
- T2ST: Data-Efficient Spatio-Temporal Foundation Models via Temporal Data and Knowledge TransferPapers · 13 Nov 2026 · Yue Xu, Kaiyun Wei, Wenying Duan, Zichen Xu, Shuai Ma, Xiaoxi He