ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVR
[Uczenie maszynowe] Autorzy: Tommy Sha, Skylar Zhai, Siqi Zhao
In reinforcement learning with verifiable rewards (RLVR) trained with group relative policy optimization (GRPO), the KL-free reward-advantage term studied here depends on within-group reward variation. If all rollouts in a group are correct or all are wrong, their group-relative advantages are identically zero; these zero-advantage silent groups provide no reward-advantage gradient, yet uniform sampling spends 39% of a run's rollouts on them. History-based prompt selection must first spend target-policy rollouts to estimate difficulty, creating a cold start with rollout waste; ThinkPrior instead uses an external anchor in one offline pass to construct a zero-rollout difficulty prior before the first target-policy rollout. The verifier-scored anchor pass rate supplies an external-anchor initialization for a Beta posterior; ThinkPrior selects by expected learnability and then updates from…
- Kategoria arxiv
- cs.LG
- Kategoria
- Uczenie maszynowe