Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This study investigates the effectiveness of large language models (LLMs) in synthesizing transformation rules for domain-specific languages, specifically targeting software evolution tasks such as API misuse correction and program migration. By evaluating three LLMs—GPT-5.4, GPT-oss-120B, and Llama3.1-8B—across diverse datasets, the research finds that GPT-5.4 consistently produces transformations that closely align with ground truth, while smaller models excel in simpler tasks but falter in complex scenarios. The findings highlight the potential of LLMs to generate effective and reusable transformation rules, although they exhibit limitations in rule applicability compared to traditional algorithms.
GPT-5.4 outperforms smaller models in synthesizing transformation rules, achieving high applicability rates while revealing the limitations of LLMs in complex software evolution tasks.
Due to their black-box nature, LLMs suffer from limited explain- ability and a lack of determinism. Their usage cost can also rise, particularly with repetitive tasks on large codebases. To mitigate this, we conduct a novel empirical study targeting three domain- specific languages for transformation rules, namely Comby, GritQL, and Ast-Grep. We evaluate three LLMs (GPT-5.4, GPT-oss-120B, and Llama3.1-8B) on six diverse datasets covering four software- evolution tasks: API misuse correction, program repair, API migra- tion, and language version migration. Our results provide evidence that transformation rule synthesis moves beyond proof-of-concept with strong frontier models. GPT-5.4 achieves consistently high rule applicability rates and produces transformations closest to the ground truth across most benchmarks. Smaller and open-weight GPT-oss-120B and Llama3.1-8B models remain effective for simpler, localized changes but struggle with complex migration scenarios. We also observe non-negligible generalizability through the usage of meta-variables and through a high reuse score in the first quartile of many datasets. Finally, when compared to the anti-unification algorithm, LLMs outperform it in correctness, but underperform in rule applicability. Overall, our results show great potential for LLMs to generate sound, correct, generalizable, and reusable rules.