Search papers, labs, and topics across Lattice.
This paper explores a novel approach to synthesizing formal specifications from test code and execution traces, circumventing the need for white-box access to source code. By leveraging LLMs to infer specifications based solely on program interfaces and dynamic outputs, the method addresses significant barriers to industrial adoption related to intellectual property and deployment costs. Initial experiments on the SpecGenBench benchmark reveal that this technique can effectively guide LLMs toward generating meaningful Java Modeling Language specifications, although challenges in checker compatibility and diagnostic feedback remain.
LLMs can infer formal specifications from tests alone, potentially transforming how we approach specification synthesis in industry.
Formal specifications offer strong guarantees, but remain costly to write manually. Recent LLM-based approaches automate this by inferring specifications from source code, yet their reliance on white-box access poses barriers to industrial adoption due to intellectual property risks and deployment costs. Our approach uses LLMs to infer candidate specifications solely from test code and dynamic execution traces: the LLM observes only the program interface, selected inputs, and corresponding outputs or state changes, while the implementation internals remain hidden. Candidate specifications are validated locally using bounded model checking, with feedback guiding iterative refinement. Initial results on the SpecGenBench benchmark suggest that tests can guide LLMs towards meaningful Java Modeling Language specifications, while also highlighting checker compatibility and diagnostic feedback as key challenges for reliable refinement.
