Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of using Large Language Models (LLMs) in generating requirements from source code, aiming to address the scarcity of adequate datasets in Requirements Engineering. Despite the promise of LLMs, the research found that they struggled to produce high-quality requirements, reliably generate non-implemented requirements, and effectively introduce synthetic requirements smells. The findings indicate that current LLM capabilities are insufficient for generating accurate code-to-requirements datasets without substantial human oversight for quality assurance.
LLMs fail to generate reliable requirements from code, revealing critical limitations in their current capabilities for empirical Requirements Engineering.
Empirical research in Requirements Engineering is hampered by a lack of adequate datasets that pair source code with corresponding requirements. A tempting route to addressing this lack is the use of Large Language Models to synthesize requirements from existing code bases. We investigate this question by evaluating an LLM-based and RAG-supported agentic approach that generates requirements from source code, verifies their implementation status relying on a human-in-the-loop, and synthetically introduces requirements smells and non-implemented requirements. Our goal was to create datasets that mimic reality and foster empirical RE research. However, during the study, various problems arose, leading to this experience report. Contrary to our initial hypotheses, LLMs were unable to (i) generate non-implemented requirements reliably, (ii) generate high quality requirements, and (iii) reliably introduce synthetic requirements smells. Furthermore, neither an LLM nor a single human-in-the-loop suffices to detect requirements smells reliably. These findings suggest that the generation of code-to-requirements datasets using LLMs is not yet viable and requires human supervision, especially for quality assurance. We critically reflect on our lessons learned and draw relevant conclusions for both researchers and practitioners.
