Search papers, labs, and topics across Lattice.
This paper introduces two large-scale datasets, AIOps2025 and RCA100, designed to evaluate LLM agents in microservice failure diagnosis through a systematic reasoning-process evaluation paradigm. Unlike existing benchmarks that focus solely on final outcomes, these datasets assess agents' diagnostic capabilities across three critical dimensions: Localization, Identification, and Reasoning, providing a comprehensive view of their performance. With over 500 expert-labeled failure cases and validation through large-scale competitions, these datasets offer a robust framework for advancing the field of AgentOps in real-world microservice environments.
Evaluating LLM agents in microservice failure diagnosis reveals that traditional outcome-based benchmarks miss critical reasoning processes, which these new datasets effectively capture.
LLM-based agents are reshaping microservice operations into AgentOps, where benchmarks are key to evaluating failure diagnosis over multimodal observability data. However, existing benchmarks remain largely outcome-oriented: they score only the final answer and fail to assess the systematic reasoning process in failure diagnosis. We address this gap by introducing two large-scale datasets (AIOps2025 and RCA100) under a reasoning-process evaluation paradigm that assesses agentic diagnostic capability along three dimensions: Localization (where the fault occurs), Identification (what type of fault it is), and Reason (whether the reasoning trace is grounded in relevant evidence). Together, the two datasets comprise over 500 expert-labeled failure cases across two representative microservice systems (HipsterShop and the OpenTelemetry Demo Store). They cover diverse fault scenarios across resource, network, runtime, middleware/database, and application-logic categories and provide fine-grained causal evidence to support agent learning and reasoning-process evaluation. Beyond scale and coverage, the datasets have been carefully labelled by domain experts and validated through large-scale competitions, supporting more than 6,000 participating teams. This makes them not only expert-labeled diagnostic datasets, but also competition-validated benchmarks for evaluating agentic failure diagnosis in real-world microservice environments. Datasets are available at https://www.aiops.cn/gitlab/aiops-live-benchmark/agenticopseval.
DAMO
Tsinghua AI