Search papers, labs, and topics across Lattice.
This paper introduces a pipeline leveraging large language models (LLMs) to streamline the identification and evaluation of software tools, specifically focusing on log anomaly detection. By screening 3,233 studies, the authors reduced the pool to 569 relevant papers and identified 83 unique tools, ultimately achieving 24 successfully running implementations with minimal human effort. The approach demonstrates significant efficiency gains in conducting rapid reviews, suggesting a scalable method for tool discovery in software engineering research.
LLMs can accelerate the discovery of functional software tools, achieving 24 successful implementations from 83 identified tools with just 4 hours of human effort.
In software engineering research, the primary outcome is frequently a tool. However, for practitioners and academics alike, it is hard to tell which tools are maintained and do they work out of the box. In this paper, we propose a pipeline to identify relevant studies with LLM screening, extract the tools presented in them, and run them with LLM-based coding agent. To evaluate the feasibility of our approach we focus on software log anomaly detection tools. We begin the study by designing a broad search string that yields 3233 hits from Scopus. We request two LLMs to provide an inclusion probability for each title-abstract pair according to the inclusion and exclusion criteria. From the 3233 exported abstracts, this screening reduced the number of included papers to 569, out of which we could download 470. These papers included 206 unique links and after manual evaluation we determined 83 to be tools. Finally, we ran the LLM-based coding agent on these 83 links, and got 24 successfully running tools. As replicating our approach would require roughly only 4 hours of human effort, of which 3 hours were manual PDF downloading, and 12 hours of LLM running time, this demonstrates promising efficiency when utilizing LLMs in rapid reviews. Because practitioner-built tools often lack academic papers, in the future we aim to expand our analysis to tool-hosting platforms such as GitHub and PyPI. In the future, we plan to formalize our workflow as LLM Agent Skills to make our approach easier to adopt.
