Search papers, labs, and topics across Lattice.
This paper investigates the reliability of using ROC curves and AUC for evaluating Software Defect Prediction (SDP) models. By decorating ROC curves with threshold values and representing True Positive Rate (TPR) and False Positive Rate (FPR) as functions of the threshold, the authors demonstrate that high AUC values do not guarantee superior performance compared to a random model across all thresholds. The study reveals that traditional evaluation criteria based solely on AUC can lead to incorrect conclusions about SDP model effectiveness.
A high AUC in software defect prediction doesn't guarantee your model actually outperforms random guessing across all decision thresholds, undermining a common evaluation practice.
Background: Receiver Operating Characteristic (ROC) curves are widely used to evaluate the performance of Software Defect Prediction (SDP) models that estimate module fault-proneness, i.e., the probability that a module is faulty. A ROC curve maps a model's performance in terms of True Positive Rate and False Positive Rate for any possible threshold set on fault-proneness. The Area Under the ROC Curve (AUC) summarizes the performance of a model across all possible thresholds. Traditionally, ROC curves completely above the bisector of the ROC space are considered better than random, and high AUC values are associated with good performance. Aim: We investigate whether these beliefs are correct, hence if SDP model evaluation based on ROC curves and AUC is reliable. Method: We decorate ROC curves by highlighting the points corresponding to threshold values. We also represent True Positive Rate and False Positive Rate as functions of the threshold. Thus, we can evaluate whether a model classifies both faulty and non-faulty modules better than the random model. Results: We show that commonly used evaluation criteria may lead to wrong conclusions. Conclusions: A high value of AUC does not guarantee that both the True Positive Rate and the False Positive Rate of a model are better than the random model's for all possible thresholds. Either decorated ROC curves or alternative representations are needed to appreciate all the relevant aspects of SDP models.
