Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of probability-based uncertainty quantification (UQ) metrics in the context of software defect prediction (SDP) by analyzing their relationships with performance and calibration across various classifiers and datasets. The findings reveal that UQ metrics are highly context-dependent, with different correlations observed between UQ and performance metrics like false positive rates and AUC in within-project defect prediction (WPDP) compared to cross-project defect prediction (CPDP). Notably, the research emphasizes the necessity of evaluating UQ against specific performance objectives and the importance of independent calibration assessment, suggesting that transferred probabilities require revalidation before being used in quality assurance decisions.
UQ metrics in software defect prediction can mislead if applied without context, revealing that strong classifiers may still suffer from significant calibration errors.
Software defect prediction (SDP) classifiers produce probabilities used for inspection prioritization, threshold tuning, and risk communication. Probability-based uncertainty quantification (UQ) characterizes prediction confidence, but whether common UQ metrics reliably indicate performance and calibration remains unclear. We conducted a large-scale empirical study of probability-based UQ for SDP. We evaluated five UQ metrics, six performance metrics, and three calibration metrics for 16 representative classifiers. We analyzed these relationships under two prediction settings: within-project defect prediction (WPDP), using 36 benchmark datasets, and cross-project defect prediction (CPDP), using 32 feature-compatible datasets. Results showed that UQ was highly context-dependent. Under WPDP, UQ correlated more consistently with false positive rate and AUC than with MCC, F1 score, and other metrics; these correlations also varied across classifier categories and dataset collections. Performance and calibration were related but not interchangeable; classifiers with strong discrimination could still exhibit large calibration error. Under CPDP, several UQ-performance and UQ-calibration correlations weakened or reversed, indicating that uncertainty signals do not reliably transfer across projects. Thus, UQ should be evaluated against specific performance objectives. Calibration should be assessed independently using multiple metrics. Transferred probabilities should be revalidated before guiding quality-assurance decisions.
