Search papers, labs, and topics across Lattice.
This paper introduces R2Act, a novel evaluation framework designed to assess the effectiveness of recovery actions in incident response for cloud-native microservice systems. By analyzing a benchmark dataset of 302 quality-audited Kubernetes incidents, the authors reveal that while state-of-the-art LLMs achieve high accuracy in diagnosing root causes, their ability to select valid recovery actions is significantly lacking, with recovery validity rates ranging from 36.8% to 60.3%. This highlights a critical gap in the current use of LLMs for operational tasks, emphasizing that the challenge lies not in diagnosis but in the translation of that diagnosis into effective recovery actions.
Despite high diagnostic accuracy, LLMs fail to choose valid recovery actions for over 60% of incidents, exposing a critical flaw in their operational utility.
Large language models (LLMs) are increasingly used to interpret operational evidence and assist incident response in cloud-native microservice systems. However, recovery-oriented use cases require more than identifying a root cause. After observing symptoms and diagnosing a fault, an operator or agent must translate the diagnosis into a concrete recovery action, apply it to an admissible target, and verify that service health has been restored. Existing RCA and log-analysis evaluations are well-suited to diagnosis, but they do not characterize this subsequent action decision. This paper presents R2Act, a recovery-action evaluation framework for post-diagnosis incident response. R2Act defines an incident schema, quality gate, action-space representation, recovery-validity metrics, offline evaluator, and live-replay protocol. We instantiate the framework as a benchmark dataset of 302 quality-audited Kubernetes incidents from \system. Each incident provides synchronized multi-modal observations, root-cause labels, an incident-specific action space, and annotated valid and invalid recovery plans. We evaluate heuristic, supervised, RCA-oriented, deep log, and LLM-based methods. The strongest RAG-based LLMs reach 91.4\%--99.7\% root-cause service accuracy, yet their recovery validity remains only 36.8\%--60.3\%. Even when both the root-cause service and fault type are correct, recovery-oriented methods still choose invalid actions for 39.5\%--62.0\% of correctly diagnosed incidents. Overall, this work reveals that many recovery failures arise not from missing diagnostic knowledge, but from the difficulty of translating diagnostic evidence into valid recovery actions and admissible targets. This work provides a reproducible, simplified starting point for research and evaluation.
Microsoft Research