Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Otap, a novel evaluation metric for agent trajectories that leverages an unbalanced fused Gromov-Wasserstein transport approach to assess the execution graphs of planning agents against valid solution graphs. By reframing trajectory evaluation as a distance measure, Otap effectively distinguishes between sound solutions and those that succeed by chance, addressing limitations of binary success flags and exact matching methods. The results demonstrate that Otap significantly outperforms traditional metrics in accurately identifying valid trajectories, especially when the dependency graph is accurately reconstructed.
Otap reveals that traditional evaluation methods fail to capture the nuances of agent planning, enabling a more robust distinction between valid and invalid trajectories.
Large language model agents solve tasks by generating trajectories that interleave planning, tool calls, and intermediate results. Current evaluation metrics reduce such a trajectory to a binary success flag or compare it against a reference by exact matching. A success flag cannot distinguish a sound solution from one that succeeds by luck, and says nothing about why a failed run went wrong. Exact matching penalizes plans that are valid but reordered or decomposed differently from the reference. We reframe trajectory evaluation as a distance between the agent's execution graph and a set of valid solution graphs, and instantiate it via an unbalanced fused Gromov-Wasserstein transport problem over attributed dependency graphs. The resulting score, termed \otap{} (Optimal Transport for Agentic Planning), is a pseudo-metric that is provably invariant to dependency-preserving reorderings and has bounded sensitivity to redundant steps. Its unbalanced marginals handle missing or hallucinated steps without forcing a match, and its soft coupling accommodates variation in plan granularity. On controlled perturbations and three public benchmarks, \otap{} separates valid from invalid trajectories in a regime where semantics-only metrics score below chance. Its accuracy is highest when the dependency graph is recovered exactly, and drops only when the graph is inferred heuristically from free-text traces.