Search papers, labs, and topics across Lattice.
This paper introduces Influence-Aware Policy Optimization (IAPO), a novel method for credit assignment in multi-turn service agents that utilizes influence-dependency graphs to analyze the flow of information and errors during agent interactions. By leveraging user and tool observations, IAPO effectively redistributes trajectory-level advantages based on the support-use and failed-use structure of agent actions. Experiments with Qwen3-4B and Qwen3-8B demonstrate that IAPO outperforms existing multi-turn reinforcement learning baselines across multiple benchmarks without sacrificing function-calling performance.
IAPO redefines credit assignment in multi-turn interactions, showing that leveraging influence-dependency graphs can significantly enhance service agent performance.
Large Language Model (LLM) agents increasingly solve long-horizon tasks through multi-turn interactions with users and external tools. In these settings, relevant task information often unfolds over time rather than being fully specified at the initial prompt. Service agents make this challenge especially concrete: users may clarify or revise their goals, while tool responses provide information needed for subsequent decisions. Thus, a final reward alone cannot indicate which actions contributed to resolving the task. Recent methods rely on comparative evidence from other trajectories or resampled continuations, or on separately constructed step-level learning signals, to refine credit. However, a completed rollout already records how information and errors flow between agent actions. We introduce Influence-Aware Policy Optimization (IAPO), which represents each rollout as a typed influence-dependency graph over trainable agent actions, with user and tool observations serving as evidence. IAPO converts support-use and failed-use structure into routing weights that redistribute the same trajectory-level advantage. Experiments with Qwen3-4B and Qwen3-8B demonstrate superior performance over multi-turn reinforcement learning (RL) baselines across three service-agent benchmarks: {蟿^2}-Bench, UserBench, and AgentChangeBench. BFCL-v4 Multi-Turn further shows that these gains do not compromise multi-turn function-calling performance. This work advances the understanding of credit assignment in multi-turn user interactions and provides a principled approach to training service agents from sparse outcome feedback.
