Rethinking On-Policy Distillation of Large Language Models II: One Training Example
[Sztuczna inteligencja] Autorzy: Zixuan Fu, Bingxiang He, Yuxin Zuo i 10 więcej
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure \emph{state coverage}, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches \(71.5\%\), most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach \(98.9\%\) and match full-data train…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja