Search papers, labs, and topics across Lattice.
This paper analyzes the reliability of using the Lean theorem prover to judge the correctness of natural language mathematical answers, finding that its accuracy is highly dependent on proof coverage and faithfulness. They introduce COVCAL, a method that uses Lean-trace diagnostics to certify a selective-risk bound on accepted answers, allowing the system to abstain when confidence is low. Experiments on MATH-500 demonstrate that COVCAL's feasibility hinges on the coverage of the autoformalizer, with a prover-specialized formalizer achieving significantly better results than a general-purpose one.
Lean-based mathematical reasoning can be surprisingly unreliable, but this work shows how to selectively trust its judgments with provable risk bounds, paving the way for more robust automated math problem solving.
Lean is increasingly used to judge natural-language mathematical answers, but its signal is partial: many answers never formalize, and a failed proof may reflect an ill-typed statement or a missing library fact, not a wrong answer. On MATH-500 we show this signal is (i) sharply coverage-dependent, that is the proof-winning answer is correct 96% of the time at high proved coverage but 20% at low, and (ii) sparse and often unfaithful: a 7B autoformalizer proves a class for only 28% of problems, and a manual audit finds only approximately 43% of those proofs faithful. We propose COVCAL, a selector over Lean-trace diagnostics that certifies a finite-sample selective-risk bound on accepted answers or abstains, under two regimes (a conservative Bonferroni bound and a tighter dev-then-cal rule). Feasibility depends on autoformalization coverage: with the 7B formalizer the signal is too sparse and Bonferroni abstains on all 20 bootstrap partitions, whereas a prover-specialized formalizer reaches 79% coverage and flips it to feasible on 17 of 20, accepting approximately 48% of problems at 0.98 accepted accuracy. Since self-consistency alone is already 91% accurate, our contribution is a precise account of when, and with which formalizer, a partial formal signal can be trusted under risk control.
