Search papers, labs, and topics across Lattice.
This study systematically evaluates the ability of 30 large language models (LLMs) to generate correct TLA+ specifications from natural language, a task critical for industrial verification. Despite achieving up to 26.6% syntactic correctness, the models only reached 8.6% semantic correctness, highlighting significant limitations in their output quality. Notably, model size did not correlate with performance, as smaller models outperformed larger ones, indicating that reasoning alignment is crucial for formal language tasks.
LLMs struggle to produce reliable TLA+ specifications, achieving only 8.6% semantic correctness, which underscores the need for expert oversight in automated formal verification.
TLA+ has supported industrial verification at companies such as Amazon and Microsoft, yet writing correct TLA+ specifications from natural language still requires time and expertise, which limits adoption. LLMs show promise, but no prior study measures whether they produce semantically correct TLA+ specifications from natural language. This paper presents the first systematic evaluation of LLM-based TLA+ specification synthesis from natural language. Our study evaluates 30 LLMs across eight families on a curated dataset of 205 TLA+ specifications: 25 open-weight models across four prompting strategies (2,600 runs) and 5 proprietary models under few-shot prompting (130 runs), all validated by the SANY parser and TLC model checker. LLMs achieve up to 26.6% syntactic correctness but only 8.6% semantic correctness, with successes exclusive to progressive prompting. Results show that model size does not predict quality, e.g., DeepSeek r1:8b outperforms its 70B variant across all strategies, which suggests the importance of reasoning alignment for formal languages. Code-specialized models consistently underperform due to negative transfer from mainstream language training. We identify five recurring hallucination categories, all traceable to specific training data biases. These results suggest that current LLMs do not generate reliable TLA+ specifications without expert oversight. We release the evaluation framework, code, and dataset to support reproducibility and future research.
