Search papers, labs, and topics across Lattice.
This paper introduces preference-based trajectory evaluation, a method that assesses agentic systems by comparing trajectories based on temporal preferences rather than solely on terminal success. This approach addresses the issue of statistical inefficiency caused by traditional success-based metrics, which often lead to tied comparisons in performance evaluations. The authors demonstrate that their method significantly reduces ties from approximately 75% to 35%, thereby enhancing discriminative power and ranking stability across various benchmarks.
Traditional success metrics tie agent performance 75% of the time鈥攖his new approach slashes that to 35%, revealing clearer distinctions in agent capabilities.
Offline evaluation of agentic systems often collapses trajectories to terminal success, discarding information about partial progress and inducing widespread ties, creating substantial statistical inefficiency by reducing effective sample size and weakening the ability to distinguish systems. We propose preference-based trajectory evaluation, which compares trajectories directly through temporal preferences over progress and time-to-return profiles. We find that, across diverse agentic and interactive benchmarks, standard success-based metrics produce tied comparisons on roughly 75% of instances, whereas trajectory-aware preferences reduce ties to roughly 35%, improving discriminative power, ranking stability, and data efficiency. Our results suggest that benchmark saturation, often attributed to poor data collection or problem difficulty, may also be explained by the choice of evaluation measure.
CMU ML