Search papers, labs, and topics across Lattice.
This paper introduces a framework for evaluating the capabilities of LLMs in generating code for constraint domain-specific languages (DSLs) from textual specifications, assessing the generated code for well-formedness and correctness. The framework is applied to OCL and Alloy, comparing performance against Python. Results indicate that LLMs perform better with Python and that smaller context windows hinder constraint-related code generation, while code repair and multiple attempts improve code quality.
LLMs struggle with niche DSLs like OCL and Alloy compared to Python, but surprisingly, simple techniques like code repair can significantly boost their performance.
Large language models (LLMs) can be used to support software development tasks, e.g., through code completion or code generation. However, their effectiveness drops significantly when considering less popular programming languages such as domain-specific languages (DSLs). In this paper, we propose a generic framework for evaluating the capabilities of LLMs generating DSL code from textual specifications. The generated code is assessed from the perspectives of well-formedness and correctness. This framework is applied to a particular type of DSL, constraint languages, focusing our experiments on OCL and Alloy and comparing their results to those achieved for Python, a popular general-purpose programming language. Experimental results show that, in general, LLMs have better performance for Python than for OCL and Alloy. LLMs with smaller context windows such as open-source LLMs may be unable to generate constraint-related code, as this requires managing both the constraint and the domain model where it is defined. Moreover, some improvements to the code generation process such as code repair (asking an LLM to fix incorrect code) or multiple attempts (generating several candidates for each coding task) can improve the quality of the generated code. Meanwhile, other decisions like the choice of a prompt template have less impact. All these dimensions can be systematically analyzed using our evaluation framework, making it possible to decide the most effective way to set up code generation for a particular type of task.
