Search papers, labs, and topics across Lattice.
This study evaluates the effectiveness of large language models (LLMs) in generating specifications for the VeriFast static verifier, focusing on 303 C functions. By employing various prompting strategies and analyzing the outputs quantitatively and qualitatively, the research reveals that while LLMs maintain high functional accuracy (over 91%), their success in verification is limited to 31.4%, primarily due to errors in domain-specific knowledge related to separation logic. The findings highlight the potential for improved LLM performance through targeted prompting and the use of formal contracts, offering insights for enhancing specification generation in static verification contexts.
LLMs generate functional specifications with over 91% accuracy, but struggle with verification success, revealing a critical gap in domain knowledge for separation logic.
Static verification tools can assure industrial scale software, but require significant human labor to write specifications. This is particularly true of static verifiers based on separation logic (SL verifiers), which excel at verifying heapmanipulating programs, but require many complex auxiliary specifications to reason about heap structure. Recent work applies large language models (LLMs) to generate code, tests, and proofs, including specifications for verifiers, but mostly targeting non-SL verifiers. To address this gap, this paper thoroughly evaluates how well LLMs perform when prompted to generate specifications for verifying 303 C functions with the SL verifier VeriFast. We explored eight prompting approaches, ten LLMs, and three input types in two stages. Quantitative and qualitative analyses are used to assess the LLM-generated code and specifications for functional behavior, verifiability and errors. The results show that LLMs preserve functional behavior in source code and specifications (both over 91%), but achieve modest verification success (31.4%). Using Gemini 2.5 Pro and providing formal contracts lead to higher success rates in our setting. Moreover, most errors (94%) come from LLMs' mistakes in the domainspecific knowledge of SL verifiers such as VeriFast. These findings provide guidance for optimizing LLM-generated specifications for SL verifiers.
