Search papers, labs, and topics across Lattice.
This paper introduces SPEAR, a novel method for reinforcement learning-based knowledge distillation that addresses the limitations of existing reward systems by providing a training-free, plug-and-play process reward mechanism. By projecting natural-language reasoning into symbolic milestones and utilizing the longest common subsequence (LCS) for alignment, SPEAR generates dense, order-aware rewards that enhance logical consistency in student models. Experimental results across various reasoning tasks show that SPEAR effectively narrows the reasoning gap between teacher and student models, demonstrating its potential for efficient sequence-level distillation.
SPEAR bridges the reasoning gap in reinforcement learning by transforming complex natural-language reasoning into efficient symbolic milestones, enabling dense and logical reward signals without costly neural verifiers.
Reinforcement learning-based knowledge distillation has the potential to transfer complex reasoning from teacher to student models, yet it currently faces a critical dilemma: researchers must choose between sparse outcome-based rewards, which provide insufficient logical guidance, or expensive neural Process Reward Models (PRMs) for dense signals. We resolve this by introducing SPEAR (Symbolic Process Evaluation and Alignment Reward), a training-free and plug-and-play process reward method for sequence-level on-policy distillation. SPEAR projects natural-language reasoning traces into domain-adaptive symbolic milestones, providing an efficient proxy for process-level reasoning alignment. By utilizing the longest common subsequence (LCS) to align student explorations with teacher milestones, SPEAR provides a dense, order-aware reward signal that enforces logical consistency without the need for an external neural verifier. Our experiments across math, science, and commonsense reasoning tasks demonstrate that SPEAR effectively bridges the reasoning gap between student and teacher models via sequence-level distillation with efficient dense process rewards. Our code and data are available at: https://github.com/zhuochunli/SPEAR.
