Search papers, labs, and topics across Lattice.
To resolve the severe critic bias induced by policy lag in asynchronous LLM reinforcement learning, this paper presents BRACE, an off-policy value correction algorithm designed for long-horizon agentic tasks. The approach overcomes the trade-off between reward detachment (from short horizons) and exploding importance-weight variance (from long horizons) by restricting Bellman corrections to an initial token prefix while anchoring an unweighted Monte-Carlo tail. On the BrowseComp-Plus benchmark, BRACE maintains training stability up to 50 updates off-policy, delivers a 2.4% boost in mean@1 over strong baselines, and runs 2.46脳 faster per step than synchronous training.
Asynchronous LLM agents can now train up to 50 updates off-policy without critic collapse by simply decoupling token-level importance correction from long-horizon reward propagation.
Asynchronous reinforcement learning has become the standard way to scale training for language models, but the resulting policy lag biases the critic toward the stale behavior policy. Existing work on asynchronous LLM training corrects the actor and leaves this bias unaddressed, while the off-policy value correction of classical RL does not carry over to long-horizon agentic tasks, since a short correction horizon leaves the regression target free of the reward and a long one lets the product of importance ratios drift exponentially with the trajectory length. We propose BRACE, an anchored Bellman-residual correction for stale value models. BRACE bounds the correction horizon to a prefix of policy tokens and anchors a constant-weight Monte-Carlo tail beyond it, which separates policy correction from reward propagation. BRACE improves mean@1 on BrowseComp-Plus by $2.4\%$ over the strongest baseline, runs $2.46\times$ faster per step than synchronous training, and remains stable $50$ updates off-policy.
