Dr. OPD: Learning What to Follow for Optimal On-Policy Distillation of Large Language Models
[Uczenie maszynowe] Autorzy: Zhenyu Wang, Tianze Wang, Linjun Zhang i 1 więcej
On-policy distillation (OPD) trains a student on its own generated responses using dense, token-level supervision from a stronger teacher. Vanilla OPD treats all teacher signals equally, assuming that the teacher's supervision is equally important for every token. However, teacher signals at different tokens may have very different effects on the student's performance: some correct important reasoning errors, while others have little effect on the final answer. Motivated by this observation, we introduce Dr. OPD (OPD Done Right), which defines the optimal weighted OPD to maximize the student's performance. We formulate Dr. OPD as a bilevel optimization problem in which the student learns from weighted teacher supervision, while the weights are selected to maximize the expected reward of the resulting student. To solve Dr. OPD, we develop an efficient iterative solver that updates the to…
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe