Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs
[Przetwarzanie języka] Autorzy: Jingtan Wang, Arun Verma, Xiaoqiang Lin i 4 więcej
How to divide a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL) during LLM post-training remains an open problem. Existing work characterizes only broad trends (e.g., SFT dominates in low-data regimes), lacks a principled allocation framework, and does not examine whether the optimal ratio transfers across model sizes. We frame this problem in terms of near-optimality: rather than seeking a single optimal SFT-RL ratio, we characterize the near-optimal region, the set of allocations within a specified tolerance of peak performance. Empirically, this region is wide even for small tolerances (2-10%), widens with model scale, and transfers reliably from small proxy models to large target models. This yields a practical strategy: small proxy-model experiments suffice to identify a transferable near-optimal region, eliminating the need for exhausti…
- Kategoria arxiv
- cs.CL
- Kategoria
- Przetwarzanie języka