Search papers, labs, and topics across Lattice.
This paper critiques the conventional approach to GPU reliability assessment, which relies on predicting the exact timing of failures, and reveals the inherent difficulties in achieving accurate predictions due to the stochastic nature of GPU failures. By analyzing telemetry data from a production cluster, the authors introduce HeaRank, a Learning-to-Rank framework that ranks GPU nodes based on their relative failure risk rather than attempting to predict specific failure times. HeaRank demonstrates superior performance, achieving an AUC of 0.83 and capturing 64% of future failures within the top 5% of ranked nodes, significantly outperforming existing systems.
Rethinking GPU reliability, this study shows that ranking nodes by failure risk can outperform traditional predictive maintenance methods, capturing 64% of failures in the top 5% of at-risk GPUs.
The reliability of Graphics Processing Units (GPUs) is a criticalbottleneck for modern large-scale AI infrastructure, where a sin-gle node failure can disrupt synchronous training jobs and causesignificant financial losses. While predictive maintenance is widelyused in other hardware domains, we demonstrate that accuratelypredicting the exact timing of GPU failures is inherently difficult.Through an in-depth analysis of telemetry data from a productioncluster, we find that major GPU failures, including Double Bit Er-rors (DBEs) and GPU Lost events, exhibit strong stochasticity andlow signal-to-noise ratios in time-series telemetry, which makesconventional time-based prediction ineffective. This insight motivates a paradigm shift: instead of attempting topredict the absolute timing of a failure, we propose a more robustapproach focused on ranking nodes by their relative failure risk. Wepropose HeaRank (Health Rank), a Learning-to-Rank (LTR) frame-work that leverages stable historical failure patterns to computea global risk ranking of GPU nodes. Evaluated on a production-scale cluster with thousands of GPUs, HeaRank achieves an AUCof 0.83, significantly outperforming both heuristic baselines andstate-of-the-art ranking algorithms. In online deployment, HeaRanksuccessfully captures 64% of future failures within the top 5% ofranked nodes, compared to only 21% by the incumbent productionsystem. These results suggest that relative risk ranking can serveas a robust alternative in environments where absolute failure pre-diction is inherently limited. Our work highlights the importanceof risk-aware scheduling and proactive resource management inmodern GPU clusters.
Tsinghua AI