Search papers, labs, and topics across Lattice.
This paper introduces TheoremBench, a benchmark designed to evaluate large language models (LLMs) on theorem proving in formal mathematics, moving beyond traditional competition-style evaluations. The benchmark includes nearly one hundred classical theorems in two formats: a plain version for single theorems and a premised version that incorporates supporting subtheorems, allowing for a deeper analysis of proof structures. Experiments reveal that while explicit premises enhance performance, current provers tend to favor simpler subtheorems and often rely on inefficient proof strategies, underscoring the need for better structural evaluation metrics in this domain.
LLMs struggle with complex theorem proving, often opting for inefficient strategies that highlight a significant gap in their reasoning capabilities.
LLMs have recently achieved strong results on formal proving benchmarks. However, existing evaluations remain heavily concentrated on competition-style problems and often fail to capture how models behave on longer, more dependency-rich mathematical developments. We introduce TheoremBench, a Lean4 benchmark designed to evaluate theorem provers beyond contest settings. The benchmark is built from nearly one hundred classical theorems and is released in two complementary forms: a plain main version containing one target theorem per instance, and a premised version that expands each theorem into a structured family of related proving tasks consisting of the main theorem together with automatically extracted supporting subtheorems. This design enables evaluation of not only whether the final theorem was proved from scratch, but also of partial progress through the internal proof structure of a theorem. Our experiments show that explicit premises substantially improve performance for Lean4-capable prover models. To provide a comprehensive evaluation, we introduce theorem-level coverage and token-efficiency metrics that expose qualitative differences in proof behavior. The results show that current provers remain strongly biased toward easy subtheorems and often solve theorems through long and inefficient tactic traces rather than compact proof plans. TheoremBench therefore provides a more fine-grained view of formal reasoning ability and highlights the importance of structural benchmark design for evaluating Lean4 theorem provers.
