Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper explores the automation of UML state machine generation from unstructured natural language requirements using LLMs, comparing GPT-4o and Claude 3.5 Sonnet with single-step, multi-step, and hybrid prompting approaches. Two novel State Machine Frameworks (SMFs) are introduced: a Structure-Driven SMF generating components sequentially and an Event-Driven SMF iteratively guided by identified events. Results show that while LLMs can generate state machines with reasonable F1-scores for states and transitions, performance on guards and actions is lacking, but a hybrid approach can improve the performance of non-reasoning LLMs.
LLMs can almost fully automate UML state machine generation, but struggle with nuanced components like guards and actions, revealing a gap that a novel hybrid approach can partially bridge.
UML state machine design is a critical process in software engineering. Traditionally, state machines are manually crafted by experienced engineers based on natural language requirements-a time-consuming and error-prone procedure. Many automated approaches exist but they require structured NL requirements. In this paper, we investigate the capabilities of current Large Language Models to fully automate UML state machine generation via specialized State Machine Frameworks (SMFs) from non-structured NL requirements. We evaluate two types of state-of-the-art LLMs using single-step and multi-step prompting approaches: a non-reasoning LLM GPT-4o and a reasoning-focused LLM Claude 3.5 Sonnet, and introduce a novel Hybrid Approach that uses the output from a Single-Prompt Baseline as an initial draft state machine, which is then refined through an SMF. In our study, two distinct SMFs are developed based on human approaches: (i) a Structure-Driven SMF, in which state machine components (states, transitions, guards, actions, etc.) are generated in sequential steps, and (ii) an Event-Driven SMF, where identified events iteratively guide state machine construction. Our experiments indicate that while LLMs demonstrate a promising ability to generate state machine models from the Single-Prompt Baseline (e.g., F1-scores of 0.90 for states and 0.75 for transitions using Claude 3.5 Sonnet), their performance is not yet fully sufficient for a fully automated solution (e.g., F1-scores of 0.23 for guards and 0.00 for actions for GPT-4o). Our proposed Hybrid Approach improves the performance of the non-reasoning LLM (GPT-4o) to a similar level as the reasoning LLM (Claude 3.5 Sonnet) but does not further improve the reasoning LLM. Our evaluation highlights both the potential and the limitations of current LLMs for automated state machine design, providing a baseline for future research in this domain.