Search papers, labs, and topics across Lattice.
The paper introduces $A^2E$, an end-to-end evaluation engine designed to systematically assess the capabilities of agent harnesses used with large language models. By employing the Agent Task Protocol (ATP) and an automatically instrumented Monitor, $A^2E$ captures standardized execution traces and evaluates harness performance across multidimensional metrics, revealing significant performance variability among model-harness combinations. The findings underscore the importance of rigorous evaluation in optimizing the co-evolution of models and harnesses for diverse tasks.
No single model-harness combination consistently outperforms others, highlighting the critical need for tailored evaluations in agent deployment.
With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains. The fast-evolving harness ecosystem has also made rigorous capability evaluation increasingly important. However, efficiently building an end-to-end, systematic, and comprehensive evaluation pipeline remains a significant challenge. To address this challenge, we introduce $A^2E$ (Agent Auditing Engine), an end-to-end evaluation engine designed for agent harnesses. $A^2E$ leverages our newly proposed Agent Task Protocol (ATP) to enable the rapid integration of evaluation tasks with different harnesses. Through an automatically instrumented Monitor, it captures and generates standardized execution traces during experiments. In the Evaluation stage, $A^2E$ systematically assesses harness capabilities using a suite of multidimensional metrics. Compared with correctness alone, these metrics provide a more fine-grained characterization of differences among harnesses in execution efficiency, tool use, task planning, and error recovery. Experiments conducted with $A^2E$ further reveal that model-harness combinations exhibit substantial performance variation across different types of tasks, and that no single combination consistently outperforms all others across every task. These findings not only demonstrate the necessity of systematic evaluation but also provide useful guidance for the co-evolving of models and harnesses. Our code is available at https://github.com/datamllab/A2E.
