Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
This paper introduces Follow the Winners (FTW), a critic-free reinforcement fine-tuning algorithm for agentic LLMs designed for stateful environments where repeated rollouts are infeasible. FTW adapts the cross-entropy method by using an ordinal filter on replay-buffer samples to achieve polynomial concentration in return order statistics, mitigating variance from long, sparsely verified trajectories. Empirical results show FTW matches GRPO and PPO on Sokoban and Search-R1 benchmarks, offering a viable alternative to value models or group rollouts by trading CPU memory for performance.
Critic-free LLM fine-tuning can now achieve state-of-the-art performance in stateful environments without costly repeated rollouts, by leveraging ordinal statistics on replay buffer samples.
Critic-free reinforcement fine-tuning (RFT) for agentic large language models is often done through GRPO-style methods, which compute a group baseline over repeated rollouts to reduce target variance. However, this setup is ill-suited to agents acting in stateful environments such as live services or security sandboxes, where repeated rollouts are impractical to obtain and aggressive updates entrench the noise of long, sparsely verified trajectories. We propose \textit{Follow the Winners} (FTW), a critic-free policy-learning algorithm that adapts the cross-entropy method to RFT, replacing group rollouts with an ordinal filter on replay-buffer samples that yields polynomial concentration in the order statistic of returns. We derive FTW through a control-as-inference lens, which also recovers GRPO and DPO as specific modelling choices, identifying GRPO as risk-neutral while DPO and FTW share a bounded risk-seeking offset that FTW controls. We identify this offset as an inherent trade-off of variance reduction through ordinal filters on samples, whereas a critic model induces a different trade-off between bias and variance. Scaled to agentic LLM post-training, FTW matches GRPO and PPO on Sokoban and Search-R1 baselines, showing a viable trade-off from a value model or group rollouts to CPU memory.