Search papers, labs, and topics across Lattice.
This paper investigates the effectiveness of LoRA fine-tuning for large language models (LLMs) in automated test case generation from natural language requirements. The study evaluates various open-source and proprietary LLMs, systematically exploring the impact of LoRA hyperparameters and introducing a GPT-4o-based automated evaluation framework across nine quality dimensions. Results show that LoRA fine-tuning significantly enhances the performance of open-source models, with a fine-tuned Mistral-8B model achieving comparable results to pre-fine-tuned GPT-4.1, thereby reducing the performance gap between proprietary and open-source models.
Forget expensive giants: LoRA fine-tuning can turn open-source LLMs into surprisingly competitive test case generators, rivaling even GPT-4.1.
Automated test case generation from natural language requirements remains a challenging problem in software engineering due to the ambiguity of requirements and the need to produce structured, executable test artifacts. Recent advances in LLMs have shown promise in addressing this task; however, their effectiveness depends on task-specific adaptation and efficient fine-tuning strategies. In this paper, we present a comprehensive empirical study on the use of parameter-efficient fine-tuning, specifically LoRA, for requirement-based test case generation. We evaluate multiple LLM families, including open-source and proprietary models, under a unified experimental pipeline. The study systematically explores the impact of key LoRA hyperparameters, including rank, scaling factor, and dropout, on downstream performance. We propose an automated evaluation framework based on GPT-4o, which assesses generated test cases across nine quality dimensions. Experimental results demonstrate that LoRA-based fine-tuning significantly improves the performance of all open-source models, with Ministral-8B achieving the best results among them. Furthermore, we show that a fine-tuned 8B open-source model can achieve performance comparable to pre-fine-tuned GPT-4.1 models, highlighting the effectiveness of parameter-efficient adaptation. While GPT-4.1 models achieve the highest overall performance, the performance gap between proprietary and open-source models is substantially reduced after fine-tuning. These findings provide important insights into model selection, fine-tuning strategies, and evaluation methods for automated test generation. In particular, they demonstrate that cost-efficient, locally deployable open-source models can serve as viable alternatives to proprietary systems when combined with well-designed fine-tuning approaches.
