Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To overcome severe reward sparsity in long-horizon LLM agent training without relying on scarce SFT demonstration data, the authors construct Feedback-Enriched Environments (FEEs) that dynamically shift from explicit action guidance to observation enrichment across training episodes. This environment-side scaffolding stabilizes RL dynamics by curbing policy entropy volatility and enables the agent to internalize exploration strategies directly into its model weights rather than leaning on inference-time context. Evaluated across Qwen3 model scales on SciWorld and BFCL using GRPO, GSPO, and DAPO, FEEs consistently outperform standard sparse-reward baselines and uncover intra-group feedback consistency as a key stability boundary.
Rather than relying on scarce human demonstrations to warm up reasoning agents, dynamically scaffolding the environment's feedback allows LLMs to autonomously crack sparse-reward, long-horizon tasks and bake exploration behaviors directly into their weights.
Large Language Models demonstrate remarkable proficiency in static reasoning, yet training them as autonomous agents through Reinforcement Learning (RL) for long-horizon tasks is often hindered by severe reward sparsity. While conventional agent-side warming up via supervised fine-tuning (SFT) can alleviate this, it is frequently limited by data scarcity and constrained exploration. To address this, we propose a paradigm shift to environment-side adaptation by constructing Feedback-Enriched Environments (FEEs). Through a pilot study, we establish a feedback design strategy that reformulates environments by transitioning from action guidance to observation enrichment during the later stages of both intra-episode exploration and inter-episode evolution. Large-scale experiments on SciWorld and BFCL benchmarks using various Qwen3 model scales and RL algorithms such as GRPO, GSPO, and DAPO demonstrate that FEEs consistently yield performance improvements over standard settings. Furthermore, our analysis reveals that training with FEEs (1) stabilizes training dynamics by reducing entropy volatility, (2) facilitates proactive state-space exploration in difficult tasks, (3) ensures the internalization of environmental guidance into policy weights rather than acting as a mere inference-time prior, and (4) identifies intra-group feedback consistency as a critical boundary for stable optimization.