Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the optimal allocation of compute resources in Reinforcement Learning (RL) post-training, focusing on whether to prioritize larger policies, extended training durations, increased rollout search, or enhanced reward feedback within a fixed FLOP budget. By employing a FLOP-accounting framework, the authors reveal that the best allocation strategies are contingent on factors such as model size, compute budget, and the nature of the reward system. The findings indicate that larger models consume more compute per token, necessitating a careful balance between model size and training allocation to maximize performance under constrained resources.
Optimal compute allocation in RL post-training can significantly impact model performance, revealing that bigger isn't always better when budgets are tight.
Reinforcement Learning (RL) post-training is increasingly used to adapt foundation models for reasoning, planning, and feedback-driven robot-learning pipelines, but constrained post-training resources are often summarized by a single total FLOP budget. We study the fixed-budget decision problem behind this practice: under the same post-training budget, should one use a larger policy, train a smaller policy longer, generate more rollout search, or spend compute on stronger reward feedback? We introduce a FLOP-accounting framework for GRPO post-training that decomposes compute into rollout/search, policy-update/learning, and reward- or feedback-model evaluation. Across LoRA-adapted Qwen2.5 policies, we find conditional allocation frontiers: the best observed allocation changes with model size, compute budget, reward system, and evaluation target. Same-FLOP model-size comparisons show that model choice and training allocation are coupled because larger policies consume more per-token compute and therefore buy fewer updates or rollouts under the same budget. Reward systems also change the accounting: rule-based rewards spend nearly all non-update compute on policy rollouts, while PRM-style feedback allocates a visible part of the budget to reward-model inference. We present RACE as a diagnostic pilot-grid protocol, not a guarantee of held-out improvement, for identifying allocation regimes before expensive validation runs; our results suggest that RL post-training papers should report total FLOPs together with how compute is divided among model size, search, learning, and feedback.