Search papers, labs, and topics across Lattice.
LabelMate is a novel LLM-driven framework designed to automate the labeling of issue reports by deriving a comprehensive, project-specific label set from historical data and assigning relevant labels to new reports without the need for pre-labeled training datasets. This approach addresses the significant manual effort and limitations of existing automated labeling methods, achieving an impressive average labeling accuracy of 89.84% across 16,500 issue reports from 30 diverse GitHub repositories. The results indicate that LabelMate not only enhances the efficiency of the labeling process but also improves community engagement through better issue categorization.
Achieving 89.84% labeling accuracy without any pre-labeled training data could revolutionize how software issues are categorized and managed.
Software users often submit issue reports to a product's issue tracking system to report defects, suggest enhancements, or raise other product-related concerns. Labeling these issue reports supports effective planning and improves community engagement. However, many issue reports remain unlabeled due to the substantial manual effort required to design an appropriate label taxonomy, then assign suitable labels from this taxonomy to new issue reports. Existing automated labeling approaches attempt to mitigate these challenges. However, they suffer from key limitations, such as extensive manual intervention, the assignment of generic labels, and a dependence on existing labeled datasets. To address these limitations, we propose LabelMate, a novel Large Language Model (LLM)-driven framework that (1) derives a comprehensive, project-specific label set from historical issue reports and (2) automatically assigns relevant labels to new issue reports without requiring any pre-labeled training data. We evaluate LabelMate on 16,500 issue reports from 30 popular and diverse GitHub repositories. Based on this dataset, our approach generates a coherent list of 275 labels and achieves an average labeling accuracy of 89.84%, a statistically significant improvement over existing generic label assigning approaches. These results demonstrate that LabelMate offers an efficient, domain-adaptive solution to streamline the issue labeling process.
