Search papers, labs, and topics across Lattice.
This paper introduces KRCA, an innovative Root Cause Analysis system tailored for hyper-scale microservice environments, addressing the challenges posed by their complexity and dynamism. By employing a multi-stage pipeline that combines API-level service isolation, a skeleton-based causal graph, and a memory-augmented multi-agent framework, KRCA effectively balances diagnostic accuracy with real-time efficiency. Experimental results demonstrate that KRCA significantly outperforms existing methods, achieving AC@1 scores of 0.88 for service localization and 0.79 for failure classification, while also reducing diagnosis time by over 77% in a production setting.
KRCA slashes diagnosis time by over 77% while achieving unprecedented accuracy in identifying root causes in hyper-scale microservice systems.
Hyper-scale microservice systems have become the standard infrastructure for large-scale Internet companies. These systems consist of numerous loosely coupled microservices that evolve independently through continuous development and deployment. Such complexity makes failures unavoidable, necessitating efficient Root Cause Analysis (RCA) to help Site Reliability Engineers (SREs) quickly localize root cause services and classify failure types. However, existing RCA methods often struggle to adapt to the extreme dynamism and massive scale of these systems. In this paper, we present KRCA, an end-to-end RCA system designed for hyper-scale microservice systems. To manage the vast search space, KRCA employs a multi-stage pipeline that begins with an API-level drilldown to isolate suspicious services. It then instantiates a skeleton-based causal graph from anomalous metrics to serve as a high-recall structural prior, before utilizing a memory-augmented multi-agent framework to verify causality and generate the final failure report. By combining structured causal constraints with multi-agent reasoning, KRCA employs balances diagnostic accuracy with the efficiency requirements of real-time production use. Experimental results show that KRCA achieves AC@1 scores of 0.88 and 0.79 for root cause service localization and failure type classification, outperforming the strongest baseline by at lease 31% in absolute gains. KRCA has been deployed in Kuaishou's production environment for over six months, reducing the average diagnosis time by 77.3%.
