Search papers, labs, and topics across Lattice.
This paper explores the integration of symbolic proof assistants, specifically Lean, into reinforcement learning from verifiable rewards (RLVR) by providing structured feedback during training. By utilizing Lean as a symbolic process oracle, the authors demonstrate that tactic-level supervision can yield denser and more reliable credit signals compared to traditional outcome-only baselines. Experimental results show significant performance improvements on theorem proving benchmarks, indicating that symbolic proof assistants can enhance RL frameworks by combining the strengths of language models with rigorous verification processes.
Tactic-level supervision from symbolic proof assistants can significantly outperform traditional outcome-only reinforcement learning approaches in theorem proving tasks.
While reinforcement learning from verifiable rewards (RLVR) typically has relied on a single binary verification signal, symbolic proof assistants in formal reasoning offer rich, fine-grained structured feedback. This gap between structured processes and unstructured rewards highlights the importance of feedback that is both dense and sound. In this work, we demonstrate that the Lean proof assistant itself can serve as a symbolic process oracle, supplying both outcome-level and fine-grained tactic-level verified feedback during training. Proof attempts are parsed into tactic sequences, and Lean's elaboration marks both locally sound steps and the earliest failing step, yielding dense, verifier-grounded credit signals rooted in type theory. We incorporate these structured rewards into a GRPO-style reinforcement learning objective with first-error propagation and first-token credit methods that balances outcome- and process-level advantages. Experiments with STP-Lean and DeepSeek-Prover-V1.5 show that tactic-level supervision outperforms outcome-only baselines in most settings, delivering improvements on benchmarks such as MiniF2F and ProofNet. Beyond empirical gains, our study highlights a broader perspective: symbolic proof assistants are not only verifiers at evaluation time, but can also act as process-level reward oracles during training. This opens a path toward reinforcement learning frameworks that combine the scalability of language models with the reliability of symbolic verification for formal reasoning.
