Search papers, labs, and topics across Lattice.
The paper investigates whether reinforcement learning with verifiable rewards (RLVR) can teach a compact language model (1.5B parameters) to reason about beam statics, a classic engineering problem. They train the model using parameter-efficient RL with binary correctness rewards from symbolic solvers, without relying on teacher-generated reasoning traces. While the best model achieves a 66.7% improvement in Pass@1 over the base model, it exhibits anisotropic generalization, succeeding with compositional changes but failing with topological shifts, suggesting the model learns solution templates rather than underlying physics.
Verifiable rewards in RL can lead to impressive performance gains in compact LLMs, but don't guarantee robust physical reasoning, instead inducing procedural solution templates.
Can reinforcement learning with hard, verifiable rewards teach a compact language model to reason about physics, or does it primarily learn to pattern-match toward correct answers? We study this question by training a 1.5B-parameter reasoning model on beam statics, a classic engineering problem, using parameter-efficient RLVR with binary correctness rewards from symbolic solvers, without teacher-generated reasoning traces. The best BeamPERL checkpoint achieves a 66.7% improvement in Pass@1 over the base model. However, the learned competence is anisotropic: the model generalizes compositionally (more loads) but fails under topological shifts (moved supports) that require the same equilibrium equations. Intermediate checkpoints yield the strongest reasoning, while continued optimization degrades robustness while maintaining reward. These findings reveal a key limitation of outcome-level alignment: reinforcement learning with exact physics rewards induces procedural solution templates rather than internalization of governing equations. The precision of the reward signal - even when analytically exact - does not by itself guarantee transferable physical reasoning. Our results suggest that verifiable rewards may need to be paired with structured reasoning scaffolding to move beyond template matching toward robust scientific reasoning.
