Search papers, labs, and topics across Lattice.
This paper introduces TurnSight, a novel turn-level hindsight self-distillation framework designed to enhance Tool-Integrated Reasoning (TIR) in large language models (LLMs) by providing denser supervision signals. By deriving supervision from execution-conditioned hindsight and constructing multiple hindsight views with varying lookahead horizons, TurnSight enables more precise credit assignment during long-horizon tasks. Experimental results across three benchmarks show that this approach significantly improves the performance of LLMs in complex reasoning scenarios, outperforming traditional trajectory-level methods.
TurnSight reveals that leveraging turn-level hindsight can dramatically enhance LLM performance in complex tool interactions, outperforming conventional reinforcement learning techniques.
Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions. However, existing reinforcement learning methods often rely on trajectory-level supervision, limiting fine-grained credit assignment in long-horizon TIR scenarios. On-policy self-distillation offers denser signals through teacher branches with privileged context, but existing approaches typically derive such context from ground-truth answers or retrieved skills, which may not reflect the states actually visited by the agent. Moreover, token-level supervision fails to capture the turn-level structure of tool interactions. To address this, we propose TurnSight, a turn-level hindsight self-distillation framework that derives supervision directly from execution-conditioned hindsight. It then constructs multiple hindsight views with different lookahead horizons and selects reliable supervision through cross-horizon directional agreement. Finally, the selected hindsight signal is normalized across sibling rollouts and used to adaptively modulate RL advantages while preserving their original optimization direction. Extensive experiments on three benchmarks demonstrate the effectiveness of TurnSight. Our codes are available at https://github.com/quchangle1/TurnSight.
