Search papers, labs, and topics across Lattice.
This paper introduces LCS-Bench, a comprehensive benchmark for theory-scale auto-formalization in computer science, addressing the challenges of consistency, faithfulness, scalability, and correctness in translating interdependent logical statements. The benchmark encompasses 327 textbook items and over 85K lines of Lean code, facilitating robust evaluation across various auto-formalization and theorem-proving tasks. Evaluation results indicate that even state-of-the-art models struggle with these tasks, achieving only 20.1% accuracy, highlighting the benchmark's rigor and the need for further advancements in this area.
State-of-the-art models barely scrape 20% accuracy on a new benchmark designed to tackle the complexities of theory-scale auto-formalization.
Auto-formalization is critical for scalable formal verification, but existing progress largely focuses on isolated statements, while theory-scale auto-formalization, which coherently translates hundreds of interdependent definitions, lemmas, and theorems, remains open due to challenges in consistency, faithfulness, scalability, and correctness. In this paper, we introduce LCS-Bench, a stand-alone, theory-scale benchmark based on Logics for Computer Science. LCS-Bench is built through a novel semi-automated agentic pipeline that leverages concept graphs, formal signature planning, issue tracking, sorry-filling with counter-example search, complemented by faithfulness review from human experts. The resulting artifact covers 327 textbook items, over 4,076 Lean declarations, and more than 85K lines of Lean code. The dataset supports broad evaluation through a data engine that automatically derives five tracks of evaluation benchmarks, measuring different aspects of auto-formalization and theorem-proving capabilities. We also introduce a novel evaluation protocol featuring definitional equivalence checkers, enabling more fine-grained and faithful assessment. Through extensive evaluation on 14 models, we demonstrate that (1) LCS-Bench is of high quality, consistent, and faithful; (2) the benchmark is challenging, with state-of-the-art models achieving only 20.1% on auto-formalization tasks; and (3) our analysis reveals key findings regarding theory-scale auto-formalization and suggests promising directions for future work.
