Search papers, labs, and topics across Lattice.
This paper explores methods for reducing the manual effort required to detect Architecture Technical Debt (ATD) in software projects. They combine keyword-based filtering to pre-screen candidate issues with active learning to prioritize the most informative samples for annotation. Using the "Breaking Ties" active learning strategy, they achieve an F1-score of 0.72 while reducing annotation effort by 49%, and further employ SHAP and LIME to explain model predictions, finding LIME preferred by experts for its clarity.
Slash your ATD labeling costs in half: active learning, combined with keyword filtering, gets you to 0.72 F1 with nearly 50% less annotation effort.
Self-Admitted Technical Debt (SATD) refers to technical compromises explicitly admitted by developers in natural language artifacts such as code comments, commit messages, and issue trackers. Among its types, Architecture Technical Debt (ATD) is particularly difficult to detect due to its abstract and context-dependent nature. Manual annotation of ATD is costly, time-consuming, and challenging to scale. This study focuses on reducing labeling effort in ATD detection by combining keyword-based filtering with active learning and explainable AI. We refined an existing dataset of 116 ATD-related Jira issues from prior work, producing 57 expert-validated items used to extract representative keywords. These were applied to identify over 103,000 candidate issues across ten open-source projects. To assess the reliability of this keyword-based filtering, we conducted a qualitative evaluation of a statistically representative sample of labeled issues. Building on this filtered dataset, we applied active learning with multiple query strategies to prioritize the most informative samples for annotation. Our results show that the Breaking Ties strategy consistently improves model performance, achieving the highest F1-score of 0.72 while reducing the annotation effort by 49\%. In order to enhance model transparency, we applied SHAP and LIME to explain the outcomes of automated ATD classification. Expert evaluation revealed that both LIME and SHAP provided reasonable explanations, with the usefulness of the explanations often depending on the relevance of the highlighted features. Notably, experts preferred LIME overall for its clarity and ease of use.
