Search papers, labs, and topics across Lattice.
This paper evaluates the performance of two automated taxonomy generation methods, TnT-LLM and CLIMB, on software engineering (SE) artifacts through a multi-perspective framework. The study reveals a trade-off between quality and efficiency, with TnT-LLM producing high-quality taxonomies similar to human-defined ones but at a significantly higher cost and complexity, while CLIMB is much faster and cheaper but less effective in capturing nuanced technical details. These insights provide a foundational understanding for researchers in SE to make informed decisions on using automated methods versus human expertise in taxonomy generation.
TnT-LLM delivers high-quality taxonomies but at a steep cost, while CLIMB offers rapid generation with lower quality鈥攈ighlighting a crucial trade-off for SE researchers.
Taxonomies provide a shared conceptual framework for organizing heterogeneous observations in software engineering (SE) research. Manually constructing such taxonomies is labor-intensive and requires annotators with expertise in the SE domain. While advances in Large Language Models (LLMs) have led to the emergence of automated taxonomy generation methods outside the SE domain, their applicability to technically complex SE artifacts remains unclear. In this experience paper, we present the first comprehensive empirical evaluation of how state-of-the-art automated methods perform on SE artifacts through a multi-perspective evaluation framework, including taxonomy quality, alignment with taxonomies defined by human experts, reliability under independent annotation, and efficiency. To support this evaluation, we systematically collect seven SE papers with publicly available artifacts and human-defined taxonomies, and conduct experiments using two automated methods (TnT-LLM and CLIMB) with five state-of-the-art LLMs. Our evaluation reveals a clear trade-off: TnT-LLM constructs high-quality taxonomies comparable to human-defined ones but incurs substantially higher cost and runtime and tends to generate overly complex taxonomies, whereas CLIMB is 15--40$\times$ faster and 8--49$\times$ cheaper but tends to score lower on quality when technical inference beyond surface-level similarity is required. These findings suggest that TnT-LLM and CLIMB can be used in practical situations in the SE domain, while researchers should first assess the complexity of the generated taxonomies and their cost using a subset of the target data to decide whether to use automated methods or human experts. Our work represents a first step toward a systematic understanding of automated taxonomy generation in SE, offering actionable insights for future research and practice.
