Search papers, labs, and topics across Lattice.
This paper introduces CTBench, a public benchmark designed to evaluate AI agents' troubleshooting capabilities in realistic telecom network operations, focusing on root cause analysis and path restoration. The authors highlight that while state-of-the-art agents excel at identifying endpoints in path-restoration tasks, they significantly underperform in root cause analysis, particularly with complex operational faults. Notably, even when agents provide correct answers, they frequently lack the necessary evidence-grounded diagnoses that are critical for practical applications, revealing a gap between performance and operational readiness.
AI agents may ace endpoint identification but falter in delivering the evidence-based diagnostics essential for real-world telecom troubleshooting.
Agents are increasingly considered for automating network operations and maintenance, where engineers must diagnose network faults, optimize configurations to enhance services, and reduce operational costs while acting under strict constraints. However, existing evaluations fail to accurately model real network characteristics or assess agents under partially observable telecom environments with diverse vendors, devices, protocols, and interfaces. In this paper, we introduce CTBench, a public benchmark for assessing whether an agent behaves like a competent telecom troubleshooting engineer. CTBench focuses on root cause analysis and path restoration. Each task is constructed by experts and annotated with rich task metadata, including golden evidence steps. CTBench uses expert-grounded metrics that evaluate both final answers and the diagnostic evidence. Experiments with representative harness-model combinations show that state-of-the-art agents perform very well at identifying endpoints in path-restoration tasks but, more generally, underperform in root cause analysis. In particular, agents struggle with interface state, link-layer, service-management, and other operational faults. Most importantly, even when agents produce plausible or correct final answers, they often fail to provide the evidence-grounded diagnoses required in operational practice. Our results further show that path restoration is generally more resource expensive, yet larger resource usage does not necessarily translate into better diagnosis.
