SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning
[Sztuczna inteligencja] Autorzy: Jialong Liu, Yuling Shi, Ning Yang i 2 więcej
Self-reflection is a powerful mechanism for credit assignment in human learning, converting sparse outcome feedback into actionable guidance. However, its potential for post-training Large Language Models (LLMs) remains underexplored. We propose Self-Reflective Policy Optimization (SRPO), a framework that internalizes this capability. SRPO enables LLMs to analyze their own completed trajectories, synthesize errors into concise "reflection patches," and use reflection-conditioned teacher scores on student on-policy rollouts as dense token-level training signals. This process effectively transforms sparse terminal supervision into dense, token-level learning signals without requiring external critics, separate reward models, or larger teacher models. We demonstrate that SRPO achieves state-of-the-art performance across mathematical reasoning and long-horizon agentic benchmarks with except…
- Kategoria arxiv
- cs.AI
- Kategoria
- Sztuczna inteligencja