Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Co-Evolving Acting and Forecasting (CoPE), a novel method for training language agents that leverages intermediate supervision from successful trajectories by training a single policy backbone to both act and forecast multi-step action sequences. CoPE addresses the limitations of existing methods that rely on single-step value or observation targets by creating a temporal ensemble of action forecasts that co-evolve with the acting policy. Across five benchmarks, CoPE significantly outperforms existing RL and predictive auxiliary target baselines in both aggregate performance and sample efficiency without introducing inference overhead.
Language agents can achieve superior sample efficiency and performance by jointly learning to act and forecast multi-step action sequences within a single policy backbone.
Training language agents is bottlenecked by the high cost of long-horizon rollouts, yet standard agentic reinforcement learning (RL) learns only from a terminal scalar reward, leaving the supervision in intermediate steps largely unused. Prior work exploits this supervision through value targets that estimate per-step success probabilities or observation targets that fit environment response. Both are hard to construct well: (1) value targets come from the same single scalar signal for each step in the trajectory, and (2) observation targets, such as web pages and terminal logs, are long and noisy and reach only one step ahead. We propose \name, which adds a third kind of multi-step action targets. On successful on-policy trajectories, \name trains the policy backbone to generate the action sequences that follows each history. Each action is also supervised when forecast from the preceding histories, forming a temporal ensemble that the shared backbone absorbs during training, and this enables acting and forecasting to co-evolve. The policy still acts one step at a time at inference, with no overhead. Across five benchmarks (ALFWorld, ScienceWorld, WebShop, $τ^2$-bench, and AppWorld), \name (1) achieves stronger aggregate performance than GRPO and RL baselines with predictive auxiliary targets across simulated environments, while improving overall performance on both realistic applications; (2) delivers up to $1.9\times$ and $2.0\times$ higher sample efficiency than ECHO and GRPO, respectively, on ScienceWorld; and (3) achieves these gains with a single policy backbone, without additional models, separate pretraining, or inference overhead. The codebase is available at https://github.com/ulab-uiuc/CoPE.