Reinforcement Learning with Visually Attributable On-Policy Self-Distillation
[Widzenie komputerowe] Autorzy: Shiao Xie, Siyu Chen, Jianwei Lv i 2 więcej
Multimodal reinforcement learning with verifiable rewards (RLVR) provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning. On-policy self-distillation (OPSD) complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view. However, the resulting corrections may conflate visual evidence with non-visual shifts induced by linguistic priors, response style, or the privileged view itself. We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. Directly distilling contaminated corrections may therefore reinforce erroneous token-level updates. To address this issue, we propose reinforcement learning with visually attribut…
- Kategoria arxiv
- cs.CV
- Kategoria
- Widzenie komputerowe