Search papers, labs, and topics across Lattice.
This paper introduces State2State, an innovative mid-training method for LLM agents that derives training objectives directly from environment interactions rather than relying on externally specified tasks or expert supervision. By converting explored environment states into training challenges, agents are tasked with reaching target states, which enhances the scalability and diversity of their training. Experiments demonstrate that State2State not only improves agent performance during the training phase but also boosts final performance and learning efficiency when used as a precursor to downstream reinforcement learning, showing potential for cross-environment generalization.
Training LLM agents without expert supervision can lead to better performance and generalization across diverse environments.
Training LLM agents commonly relies on supervised fine-tuning from expert trajectories or online reinforcement learning over human-specified tasks with handcrafted verifiers. Though effective, both remain bottlenecked by externally specified tasks and supervision signals, limiting the scalability and diversity of agent training. We study an environment learning paradigm in which agents acquire interaction and manipulation capabilities solely through environment interaction, without externally specified tasks. We propose State2State, an environment-derived mid-training method that converts explored environment states into training objectives, challenging agents to reach a specified target state. By deriving tasks from environment exploration and verifying success through rule-based state matching, State2State provides scalable and verifiable training objectives without expert supervision or manual task design. Experiments on ALFWorld and ScienceWorld show that State2State improves agent performance as a standalone environment-learning stage in most settings. As initialization for downstream RL, it further improves final performance and learning efficiency, with promising evidence of cross-environment generalization.
Tsinghua AI