Search papers, labs, and topics across Lattice.
This paper introduces a novel pipeline for verifying LLM-generated mathematical solutions, addressing the limitations of answer-checking methods. The pipeline uses prompts to elicit solutions in a specific format suitable for verification with proof assistants and smaller models. Experiments on multiple datasets demonstrate a low false positive rate, suggesting the pipeline's effectiveness in accurately assessing LLM mathematical reasoning.
LLMs can now have their math solutions rigorously checked by a new pipeline that uses proof assistants and structured prompting, reducing reliance on answer-only correctness.
With the growing popularity of Large Reasoning Models and their results in solving mathematical problems, it becomes crucial to measure their capabilities. We introduce a pipeline for both automatic and interactive verification as a more accurate alternative to only checking the answer which is currently the most popular approach for benchmarks. The pipeline can also be used as a generator of correct solutions both in formal and informal languages. 3 AI agents, which can be chosen for the benchmark accordingly, are included in the structure. The key idea is the use of prompts to obtain the solution in the specific form which allows for easier verification using proof assistants and possible use of small models ($\le 8B$). Experiments on several datasets suggest low probability of False Positives. The open-source implementation with instructions on setting up a server is available at https://github.com/LogicEnj/lean4_verification_pipeline.
