Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the impact of few-shot prompting on the quality of LLM-generated unit tests, exploring different sources of test artifacts (human, SBST, LLM) as examples. The study evaluates the generated tests based on correctness, coverage, readability, cognitive complexity, and maintainability using GPT-4o on HumanEval and ClassEval datasets. Results demonstrate that few-shot prompting enhances test quality, with human-written examples leading to the highest coverage and correctness, and that similarity-based example retrieval further improves prompt effectiveness.
Human-written examples in few-shot prompts supercharge GPT-4o to generate unit tests with superior coverage and correctness compared to LLM- or SBST-generated examples.
Unit testing is essential for verifying the functional correctness of code modules (e.g., classes, methods), but manually writing unit tests is often labor-intensive and time-consuming. Unit tests generated by tools that employ traditional approaches, such as search-based software testing (SBST), lack readability, naturalness, and practical usability. LLMs have recently provided promising results and become integral to developers'daily practices. Consequently, software repositories now include a mix of human-written tests, LLM-generated tests, and those from tools employing traditional approaches such as SBST. While LLMs'zero-shot capabilities have been widely studied, their few-shot learning potential for unit test generation remains underexplored. Few-shot prompting enables LLMs to learn from examples in the prompt, and automatically retrieving such examples could enhance test suites. This paper empirically investigates how few-shot prompting with different test artifact sources, comprising human, SBST, or LLM, affects the quality of LLM-generated unit tests as program comprehension artifacts and their contribution to improving existing test suites by evaluating not only correctness and coverage but also readability, cognitive complexity, and maintainability in hybrid human-AI codebases. We conducted experiments on HumanEval and ClassEval datasets using GPT-4o, which is integrated into GitHub Copilot and widely used among developers. We also assessed retrieval-based methods for selecting relevant examples. Our results show that LLMs can generate high-quality tests via few-shot prompting, with human-written examples producing the best coverage and correctness. Additionally, selecting examples based on the combined similarity of problem description and code consistently yields the most effective few-shot prompts.