Search papers, labs, and topics across Lattice.
This paper introduces AgentCompass, an open-source evaluation infrastructure designed to streamline the assessment of LLM-based autonomous agents. By organizing the evaluation process into three independent components鈥擝enchmark, Harness, and Environment鈥擜gentCompass enhances flexibility and reproducibility while minimizing redundant engineering efforts. Key results include support for over 20 benchmarks across five capability dimensions, alongside tools for diagnosing complex failure modes, which collectively advance the state of agent research.
A unified evaluation framework that simplifies the assessment of LLM-based agents could drastically enhance reproducibility and accelerate research breakthroughs.
As Large Language Models (LLMs) evolve into autonomous agents, the need for unified evaluation infrastructure becomes critical. However, current evaluation pipelines remain highly fragmented and tightly coupled, hindering reproducibility and causing redundant engineering. To address this, we introduce AgentCompass, an open-source, lightweight, and extensible infrastructure for evaluating LLM-based agents. AgentCompass organizes the evaluation process around three independent components, namely Benchmark, Harness, and Environment, thereby enabling flexible configurations without requiring the reimplementation of complex execution logic. Furthermore, it features a fault-tolerant asynchronous runtime and comprehensive trajectory analysis tools to transparently diagnose nuanced failure modes like reward-hacking. Natively supporting over 20 benchmarks across five capability dimensions, AgentCompass provides the community with a scalable and reproducible infrastructure for advancing agent research.
Tsinghua AI