Search papers, labs, and topics across Lattice.
The paper introduces a framework for evaluating AI agent reliability beyond standard accuracy metrics, focusing on consistency, robustness, predictability, and safety. They propose twelve concrete metrics to quantify these dimensions and provide a holistic performance profile. Applying these metrics to 14 agentic models across two benchmarks reveals that capability gains have not translated into proportional improvements in reliability, highlighting persistent limitations in deployed agents.
Despite progress in AI agent capabilities, reliability across crucial dimensions like consistency and robustness remains stubbornly low, revealing a critical gap in current evaluation practices.
AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many agents still continue to fail in practice. This discrepancy highlights a fundamental limitation of current evaluations: compressing agent behavior into a single success metric obscures critical operational flaws. Notably, it ignores whether agents behave consistently across runs, withstand perturbations, fail predictably, or have bounded error severity. Grounded in safety-critical engineering, we provide a holistic performance profile by proposing twelve concrete metrics that decompose agent reliability along four key dimensions: consistency, robustness, predictability, and safety. Evaluating 14 agentic models across two complementary benchmarks, we find that recent capability gains have only yielded small improvements in reliability. By exposing these persistent limitations, our metrics complement traditional evaluations while offering tools for reasoning about how agents perform, degrade, and fail.
