Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the reliability of LLMs for design synthesis, specifically UML class diagram generation, by evaluating their adherence to object-oriented design principles under various prompting strategies. The study introduces a preference-based few-shot prompting method to guide LLMs toward designs that satisfy object-oriented principles. Results show that while preference-based prompting improves adherence to design intent, it doesn't eliminate non-determinism, and model choice significantly impacts design reliability.
LLMs struggle to reliably synthesize software designs even with advanced prompting techniques, revealing a significant gap between syntactic correctness and meaningful design generation.
Large Language Models (LLMs) are increasingly applied to automate software engineering tasks, including the generation of UML class diagrams from natural language descriptions. While prior work demonstrates that LLMs can produce syntactically valid diagrams, syntactic correctness alone does not guarantee meaningful design. This study investigates whether LLMs can move beyond diagram translation to perform design synthesis, and how reliably they maintain design-oriented reasoning under variation. We introduce a preference-based few-shot prompting approach that biases LLM outputs toward designs satisfying object-oriented principles and pattern-consistent structures. Two design-intent benchmarks, each with three domain-only, paraphrased prompts and 10 repeated runs, are used to evaluate three LLMs (ChatGPT 4o-mini, Claude 3.5 Sonnet, Gemini 2.5 Flash) across three modeling strategies: standard prompting, rule-injection prompting, and preference-based prompting, totaling 540 experiments (i.e. 2x3x10x3x3). Results indicate that while preference-based alignment improves adherence to design intent it does not eliminate non-determinism, and model-level behavior strongly influences design reliability. These findings highlight that achieving dependable LLM-assisted software design requires not only effective prompting but also careful consideration of model behavior and robustness.