Search papers, labs, and topics across Lattice.
This paper introduces HIPPO, a novel reinforcement learning framework designed to mitigate the detrimental effects of data overlap in large language models (LLMs) that lead to shortcut reasoning. By employing hint-injected aggregation and a tailored pairwise reward model, HIPPO enables the model to differentiate between genuine reasoning and fabricated justifications, resulting in improved reasoning capabilities. Experimental results indicate that HIPPO significantly outperforms standard baselines and effectively generalizes to tasks outside the training distribution, extracting authentic reasoning skills rather than superficial patterns.
HIPPO reveals that hint-injected pairwise aggregation can effectively eliminate shortcut reasoning in LLMs, leading to more authentic and transferable reasoning skills.
While reinforcement learning (RL) significantly enhances LLM reasoning, its efficacy is severely undermined by Pre-RL data overlap, where RL datasets overlap with pretraining or SFT corpora, causing models to exploit shortcuts by memorizing correct answers and fabricating post-hoc reasoning. To address this, we introduce HIPPO, a novel RL framework that integrates hint-injected aggregation with a tailored pairwise reward model. By utilizing hint injection to deliberately trigger overlap-induced behaviors, the resulting traces naturally serve as explicit anchors for pairwise comparison. This provides highly discriminable preference signals, enabling a lightweight judge model to reliably distinguish genuine reasoning deduction from shortcut-driven rationalization, while the pairwise formulation ensures stable and robust optimization compared to standard PRMs. Extensive experiments demonstrate that HIPPO yields substantial improvements over standard baselines and generalizes effectively to out-of-distribution general tasks, showing it extracts authentic, transferable reasoning skills rather than superficial shortcut patterns.
