Search papers, labs, and topics across Lattice.
This paper addresses the structural failures in physics reasoning of small language models by introducing a step-level reward framework that identifies and corrects initial reasoning errors through targeted structured feedback. The method employs policy gradient training with KL regularization, allowing the model to revise its solutions without relying on ground truth solutions as targets. The results show significant accuracy improvements across five physics benchmarks, with reductions in calculation and comprehension errors, highlighting the framework's effectiveness in enhancing reasoning capabilities in constrained models.
Targeted feedback can slash calculation errors in small language models from 56.9% to 23.5%, revolutionizing their physics reasoning abilities.
Physics reasoning fails structurally in small language models: an error at any step propagates forward, corrupting every inference that follows. Limited domain knowledge, hallucination under multi-step derivation, and distributional sensitivity compound this failure. We propose a step-level reward framework that identifies the first reasoning error, generates targeted structured feedback, and trains the model to revise its solution via policy gradient with KL regularization, without exposing it to ground truth solutions as generation targets. Unlike annotation-dependent step-level methods, no preference data construction is required and the external verifier operates exclusively at training time. Across five physics benchmarks, our framework delivers accuracy gains of 17-20% over CoT prompting and 10-16% over the strongest baseline, reduces calculation errors from 56.9% to 23.5%, and reduces miscomprehension errors from 22.3% to 12.0% in the best observed cases. Conceptual errors reduce from 89.7% to 68.7%, yet persist as the hardest failure mode across all conditions.
Microsoft Research