Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces FVSpec, a benchmark designed to evaluate AI models on real-world formal software verification tasks by translating property-based tests (PBTs) from Python into Lean 4 specifications. The authors scraped over 11,000 PBTs and successfully converted 2,772 of them into 9,415 Lean specifications, addressing the complexities of modeling Python semantics and dependently-typed programming. Their three-agent LLM pipeline not only facilitates this translation but also establishes baselines for proof generation, aiming to enhance AI-assisted formal verification as software complexity grows.
Translating real-world property-based tests into Lean specifications reveals significant challenges and opens new avenues for AI-assisted formal verification of software.
We present a benchmark for evaluating AI models and agents on real-world formal software verification tasks. We first scrape 11,039 property-based tests (PBTs) from real-world Python repositories, then automatically translate 2,772 of them (25%) into 9,415 Lean 4 specifications with sorry placeholders (about 3 formalizations/PBT; we retain multiple attempts when none dominates on quality metrics). Translating PBTs into Lean specifications is challenging: it requires modeling Python semantics in Lean, inferring the logical property encoded in an imperative PBT, and handling the inherent difficulties of dependently-typed programming in a seldom-used language. We describe a three-agent LLM pipeline for transpiling PBTs into Lean specifications, evaluate coverage and quality metrics, and provide baselines for proof generation using several automated and model based approaches. All code (scraper and agents) and data (PBTs and Lean specifications) are open source. Our benchmark aims to drive progress on the underexplored problem of AI-assisted formal verification of real-world software, which is of increasing interest as AI produces more and more of the world's code.