Search papers, labs, and topics across Lattice.
This paper introduces Hindsight Supervised Learning (HSL), a novel approach that leverages unintended yet successful goals from agent rollouts to enhance the training of large language model (LLM) agents in partially observable, long-horizon settings. By relabeling completed trajectories with the natural-language goals achieved by the agent, HSL effectively provides additional supervision, which is further refined through techniques like irrelevant-action masking and sample reweighting. Experimental results demonstrate that HSL significantly improves performance on both supervised fine-tuning (SFT) and direct policy optimization (DPO), particularly excelling in long-horizon tasks with diverse goal spaces while being sample-efficient.
Hindsight Supervised Learning transforms agent rollouts into a rich source of supervision, achieving superior performance with only a fraction of the required demonstrations.
Large language model agents operate in partially observable, long-horizon settings where obtaining supervision remains a major bottleneck. We address this by utilizing a source of supervision overlooked in existing post-training methods: unintended yet successful goals embedded within agent rollouts. Specifically, we introduce Hindsight Supervised Learning (HSL), where an auxiliary LLM reviews each completed trajectory and relabels it with all of the natural-language goals the agent actually achieved. HSL then pairs the trajectory with its relabeled goals and uses these pairs for additional fine-tuning. To mitigate suboptimality in the relabeled data, we propose two learning techniques for HSL, irrelevant-action masking and sample reweighting. Our experiments show that HSL is flexible and compatible with existing post-training pipelines. It improves both SFT and DPO, with larger gains on long-horizon tasks with more diverse goal spaces. Moreover, HSL is sample-efficient: on ALFWorld, it surpasses baselines trained on the full dataset while using only one quarter of the ground-truth demonstrations.
