Search papers, labs, and topics across Lattice.
This paper introduces LLM4MTLs, an automated workflow designed to enhance the generation and evaluation of model transformation languages (MTLs) using large language models (LLMs). By systematically exploring various prompting strategies鈥攕uch as few-shot prompting, grammar prompting, and the inclusion of helper methods鈥攖he authors evaluate the syntactic and semantic quality of generated MTL code across four different languages. The findings reveal that while few-shot prompting significantly improves syntactic quality, the impact on semantic correctness is variable and heavily dependent on the specific MTL being used.
Few-shot prompting boosts syntactic quality of LLM-generated model transformation languages, but semantic correctness remains a challenge.
Model transformation languages (MTLs) are domain-specific languages for transforming models conforming to a given metamodel into other models, including textual models such as source code. Developing correct model transformations is challenging, requiring both language-specific and domain knowledge, and motivating the use of large language models (LLMs) for MTL code generation. However, due to limited training data and executable examples, LLM-generated MTL code is often not syntactically valid or semantically usable out of the box. This paper presents LLM4MTLs, an automated workflow for constructing and comparing prompting strategies for LLM-generated MTL code, together with an evaluation suite and an empirical evaluation. The workflow systematically explores prompt constructions combining few-shot prompting, grammar prompting, and helper method inclusion, and evaluates them using syntactic and semantic metrics. We construct an evaluation suite spanning four MTLs (ATL, ETL, QVTo, and the Reactions language) with executable reference scripts and manually written test suites, and evaluate across three LLMs. We find that few-shot prompting consistently improves syntactic quality across all four MTLs while gains in semantic correctness are uneven and language-dependent. For ATL, Pass@1 remains unchanged across all strategies and models, indicating that few-shot prompting improves surface-level syntax more readily than deep transformation semantics. Grammar prompting stabilizes code generation when combined with few-shot examples, but in isolation it can be ineffective or even counterproductive for certain model-language combinations. Including helper methods as a complementary amplifier can also be beneficial. Finally, LLM choice influences syntactic correctness and similarity for certain MTLs, particularly ETL and QVTo, while its influence on semantic correctness remains limited.
