Search papers, labs, and topics across Lattice.
This paper evaluates GPT-4, Claude 3, and Gemini for automated Behaviour-Driven Development (BDD) scenario generation using a newly constructed dataset of 500 user stories and requirements. The study employs a multi-dimensional evaluation framework including text/semantic similarity, LLM-based evaluation, and human expert assessment. Results show that Claude 3 generates scenarios rated highest by human experts, despite GPT-4 achieving higher text/semantic similarity scores, and that prompt engineering effectiveness is model-specific.
Claude 3 beats GPT-4 in generating high-quality BDD scenarios as judged by humans, even though GPT-4 scores higher on traditional text similarity metrics.
This paper presents an evaluation of three LLMs, GPT-4, Claude 3, and Gemini, for automated Behaviour-Driven Development (BDD) scenarios generation. To support this evaluation, we constructed a dataset of 500 user stories, requirement descriptions, and their corresponding BDD scenarios, drawn from four proprietary software products. We assessed the quality of BDD scenarios generated by LLMs using a multidimensional evaluation framework encompassing text and semantic similarity metrics, LLM-based evaluation, and human expert assessment. Our findings reveal that although GPT-4 achieves higher scores in text and semantic similarity metrics, Claude 3 produces scenarios rated highest by both human experts and LLM-based evaluators. LLM-based evaluators, particularly DeepSeek, show a stronger correlation with human judgment than with text similarity and semantic similarity metrics. The effectiveness of prompting techniques is model-specific: GPT-4 performs best with zero-shot, Claude 3 benefits from chain-of-thought reasoning, and Gemini achieves optimal results with few-shot examples. Input quality determines the effectiveness of BDD scenario generation: detailed requirement descriptions alone yield high-quality scenarios, whereas user stories alone yield low-quality scenarios. Our experiments indicate that setting temperature to 0 and top_p to 1.0 produced the highest-quality BDD scenarios across all models.
