Search papers, labs, and topics across Lattice.
This paper addresses the challenge of allocating a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL) for large language models (LLMs). By characterizing the near-optimal region of SFT-RL ratios, the authors demonstrate that this region is broad and transferable from small proxy models to larger target models, allowing for efficient budget allocation without exhaustive searches. Their findings reveal that even small proxy-model experiments can effectively identify optimal allocation strategies, which are consistent across various tasks and model families.
Small proxy models can reveal near-optimal SFT-RL budget allocations for large LLMs, streamlining the post-training process significantly.
How to divide a fixed annotation budget between supervised fine-tuning (SFT) and reinforcement learning (RL) during LLM post-training remains an open problem. Existing work characterizes only broad trends (e.g., SFT dominates in low-data regimes), lacks a principled allocation framework, and does not examine whether the optimal ratio transfers across model sizes. We frame this problem in terms of near-optimality: rather than seeking a single optimal SFT-RL ratio, we characterize the near-optimal region, the set of allocations within a specified tolerance of peak performance. Empirically, this region is wide even for small tolerances (2-10%), widens with model scale, and transfers reliably from small proxy models to large target models. This yields a practical strategy: small proxy-model experiments suffice to identify a transferable near-optimal region, eliminating the need for exhaustive large-scale search. Our results hold consistently across tasks, model families, and both preference-based off-policy and reward-supervision on-policy RL methods. We further analyze how the asymmetry in annotation costs between SFT and RL data shifts the near-optimal region.
MIT CSAIL