Search papers, labs, and topics across Lattice.
This study evaluates the challenges of faithful natural-language-to-Lean formalization, highlighting that traditional compilation metrics can mask significant gaps in semantic accuracy. By analyzing a 400-entry benchmark across various mathematical domains, the authors reveal that while a tool-augmented agent achieves a high compilation rate of 89.5%, it only maintains a consensus faithfulness of 60.5%, indicating a substantial divide between formal validity and semantic fidelity. The research underscores the necessity of distinguishing between formal validity and faithful statement generation in evaluating theorem-proving models, as evidenced by targeted human audits confirming the reliability of their proposed metrics.
A tool-augmented agent achieves high compilation rates but reveals a shocking 29-point gap in semantic faithfulness, challenging the reliability of existing evaluation metrics.
Theorem-proving benchmarks evaluate proof search against fixed formal statements, but natural-language-to-Lean formalization must generate the formal statement itself. In this setting, compilation is only a validity check: a Lean declaration may type-check while omitting hypotheses, changing domains, or expressing a vacuous claim. We study faithful statement formalization as both an evaluation problem and a bottleneck-attribution problem. On a 400-entry graduate-level benchmark spanning real analysis, complex analysis, topology, and algebra, our protocol combines Lean compilation, cross-model semantic judging, and human expert calibration. The resulting picture is different from compile-rate evaluation: a full tool-augmented agent reaches 89.5% compilation but only 60.5% consensus faithfulness, exposing a 29.0-point compile-pass but consensus-unfaithful gap. Targeted human audits support the metric as a conservative decision boundary: across available case-level audits, 96.0% of consensus-positive outputs are human-confirmed faithful, while 82.4% of compile-pass consensus-negative outputs are human-confirmed semantic failures. Under this metric, existing one-shot formalizer models and prover-oriented Lean models remain low, suggesting that formal validity, proof-oriented Lean competence, and faithful statement generation should be reported separately. We then use a full $2^3$ factorial design to decompose three recurring interventions in formalization pipelines: parametric expert drafting, Mathlib/context search, and Lean elaboration feedback. Elaboration feedback is the largest validity intervention, but it also exposes a larger compile-pass semantic-failure bucket; search mainly improves grounding and selectivity; and fine-tuned drafting is largely substitutable in this tool stack once feedback and grounding are available.
