Search papers, labs, and topics across Lattice.
This paper introduces RP-OPSD, a novel approach to multilingual reasoning transfer that enhances on-policy self-distillation by focusing on reasoning pivots鈥攌ey decisions that influence the reasoning process. By leveraging the distributional shift between teacher models with and without English reference solutions, RP-OPSD effectively prioritizes critical reasoning signals, leading to improved performance across 17 languages on mathematical reasoning tasks. The results demonstrate that this method significantly outperforms existing multilingual reasoning baselines and OPSD variants, highlighting the importance of targeted distillation in enhancing cross-lingual reasoning capabilities.
Concentrating distillation on reasoning pivots boosts multilingual reasoning performance, outperforming traditional methods across 17 languages.
Multilingual reasoning transfer is crucial for extending reasoning capabilities of large language models (LLMs) beyond high-resource languages. On-policy self-distillation (OPSD) and its variants have emerged as a promising paradigm, providing dense token-level supervision on student-generated rollouts, yet their objectives do not explicitly prioritize reasoning signals most critical to cross-lingual transfer. We characterize that target-language reasoning comprises the generation of both surface text and reasoning pivots, which are decisions that advance or redirect the reasoning process and shape subsequent inference. This motivates concentrating privileged distillation around such pivots. We therefore propose RP-OPSD, Reasoning-Pivot-guided On-Policy Self-Distillation, using the distributional shift between matched teacher views with and without an English reference solution as an operational proxy to guide privileged distillation and reference anchoring. Experiments on mathematical reasoning benchmarks covering 17 languages and multiple difficulty levels show that our method outperforms strong multilingual reasoning baselines and OPSD variants. Further analysis reveals that RP-OPSD concentrates privileged distillation on reasoning-control and problem-condistioned state-update tokens, while downweighting it for tokens that mainly support surface realization. Our code is available at https://github.com/NJUNLP/RP-OPSD.
