Strong Teacher Not Needed? On Distillation in LLM Pretraining

by [unverified] · free · Last verified 2026-06-21T03:07:56.505Z

Knowledge distillation generally assumes a strong-to-weak relationship where stronger teachers yield better students. In this work, we examine this assumption about distillation in large language model pretraining. By varying architecture sizes and training token budgets, we create strong-to-weak...