Search papers, labs, and topics across Lattice.
This paper introduces Self-Fix Step-DPO (SFS-DPO), a two-stage reinforcement learning framework designed to enhance self-correction in large language models (LLMs) through step-level reasoning and verification. The first stage optimizes step-level preferences, while the second stage trains models to self-verify and correct their outputs, with a teacher-assisted variant (SFS-DPO-R) leveraging explanatory rationales for improved error correction. Evaluations show that both SFS-DPO and SFS-DPO-R significantly outperform existing baselines, leading to higher self-correction rates and more effective reasoning in LLMs.
Self-correction in LLMs can be dramatically improved by reinforcing step-level reasoning, achieving higher accuracy and reliability in outputs.
Achieving effective self-correction, where models verify and correct their own mistakes, remains a fundamental challenge for large language models (LLMs). In this work, we propose Self-Fix Step-DPO (SFS-DPO), a reinforcement learning based, two-stage framework for step-level self-verification and self-correction. The first stage strengthens step-level reasoning via step-level preference optimization, while the second stage explicitly trains models to self-verify and self-correct. We further introduce a teacher-assisted variant, SFS-DPO-R, which incorporates explanatory rationales for error verification to provide stronger corrective signals. Comprehensive in-domain and out-of-domain evaluations across multiple LLMs demonstrate that SFS-DPO and SFS-DPO-R consistently outperform prior step-level training baselines. Our analysis further reveals improvements in self-correction frequency and effectiveness, highlighting the importance of strengthening step-level reasoning for robust performance.
