EliteReplay: Preserving Diverse Solutions in RL Post-Training via Quality-Diversity Search
[Sztuczna inteligencja] Autorzy: Hanane Nour Moussa, Doeun Lee, Sachin Kumar
Reinforcement learning has become a common post-training technique to improve the accuracy of LLMs on verifiable tasks. However, such training can narrow the set of solutions the model can produce. Several approaches have attempted to address this behavior-sharpening effect, but they often optimize metrics that do not reflect the diversity of correct solutions. Ideally, we want a model that can produce multiple good solutions, not only a single reliable one. We propose EliteReplay, a post-training recipe that pairs GRPO with an "elite" archive built by quality-diversity search. Before training, we use the frozen base model to collect solutions that span distinct approaches; during training, GRPO is combined with a replay loss on this archive. The result is a post-trained model that improves accuracy while preserving more of the base model's solution diversity. On multiple text-to-SQL ta…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja