Early-Signal Detector

method · first seen 4 Oct 2026

knowledge distillation

Also: KD, distillation, KL distillation, LLM distillation, logit distillation, model distillation, online distillation, teacher distillation

Training a smaller student model to imitate a larger teacher model's outputs or intermediate reasoning.

Knowledge distillation trains a smaller student model to match a larger teacher's soft outputs, logits, features or reasoning traces. The sources span many routine applications: a sparse on-policy distillation method for LLM reasoning traces, KD from imperfect teachers in low-resource audio, joint pruning and distillation for audio deepfake detection, shadow removal, a study of whether matrix-structured optimizers help KL distillation of Qwen students, and a repo for black-box LLM distillation via teacher API, rejection sampling, SFT and GRPO. The volume is almost entirely arXiv papers that use distillation as one component among others. The weekly counts are already falling, which fits steady background usage rather than a new idea gaining ground.

Distillation has been a standard compression and transfer technique since 2015. The narrower variants seen here, such as on-policy and black-box LLM distillation, are the parts worth tracking, but the umbrella concept is not new.

Analyst note (established), 17 Nov 2026.

Mentions per week

06619 May11 Aug10 NovPapersCodeDiscussion

PapersCodeDiscussion

Rank 38 on 14 Nov 2026 · 178 mentions in 28 days · 174 independent groups · papers → discussion → code · first seen 6 weeks ago · acceleration -0.40

Earliest sources

Latest sources

Mentioned together with

on-policy distillationlow-rank adaptationstructured pruningdiffusion modelsspeculative decodingcatastrophic forgettingfederated learningmixture of expertssmall language modelgroup relative policy optimization