Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
[Uczenie maszynowe] Autorzy: Wei Chen, Yubing Wu, Zhou Xu i 9 więcej
Reinforcement learning with verifiable rewards (RLVR) trains language models from checked task outcomes, but assigning a terminal reward to individual reasoning tokens or actions remains difficult. Group-relative methods such as Group Relative Policy Optimization (GRPO) sample several attempts per prompt, but lose their reward-based learning signal when all attempts receive the same outcome. We propose Hindsight-Privileged RLVR (HP-RLVR), a single-rollout method that turns a completed trajectory's own future into token-level update weights. The generating policy snapshot scores each eligible token under its causal history and under a hindsight context containing the realized future and verifier feedback. A normalized Jensen--Shannon divergence between the two views weights the reward relative to a historical baseline, yielding bounded token credit for both successful and failed attempts…
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe