Search papers, labs, and topics across Lattice.
This study investigates the performance gap in fault diagnosis techniques for deep learning programs, revealing that existing methods assessed via within-program cross-validation fail to generalize to unseen programs. Using DynFault, a dataset of 5,542 fault-injected training traces from 38 real-world DL programs, the authors quantified a significant gap of 0.190 in balanced accuracy between these evaluation methods. The findings highlight that program-level structure in features contributes to this gap, with curvature features proving effective for instability detection in unseen programs, while optimizer features are only beneficial for previously encountered programs.
Existing fault diagnosis techniques miss a critical 0.190 accuracy gap when applied to unseen deep learning programs, revealing a fundamental flaw in current evaluation strategies.
Deep Learning (DL) programs can fail during training for many reasons, and diagnosing the cause is a costly and time-consuming maintenance task. Techniques for diagnosing such failures are commonly assessed using within-program cross-validation, which may be inadequate for deployment settings involving previously unseen programs. It is therefore necessary to assess how performance differs across these settings and to identify the causes of any performance gap in established fault diagnosis techniques for DL. We investigate this gap using DynFault, a corpus of 5,542 fault-injected training traces from 38 real-world DL programs. We found a gap of 0.190 in balanced accuracy for existing fault diagnosis techniques between within-program evaluation and holding out whole programs. We also found the gap comes from program-level structure in the features, which led us to examine two runtime feature sets, curvature features and optimizer features, and their behavior on unseen programs. We found that curvature features are useful for instability detection on unseen programs, while optimizer and activation features help only on programs seen during training.
