Search papers, labs, and topics across Lattice.
This paper introduces Echo-GRPO, a novel framework that enhances reasoning distillation in video language models by allowing them to paraphrase privileged reasoning traces into their own idiolect. By addressing the limitations of traditional Group Relative Policy Optimization (GRPO), which restricts learning to on-policy reasoning skills, Echo-GRPO enables models to learn more advanced reasoning capabilities without the detrimental effects of gradient clipping. The approach shows consistent improvements across multiple multimodal LLM backbones and benchmarks, indicating its effectiveness in optimizing reasoning performance.
Idiolectal paraphrasing transforms how models learn reasoning by allowing them to express complex thoughts in their own unique language, leading to significant performance gains.
Recent large language models achieve strong performance on complex reasoning tasks, where reinforcement learning with Group Relative Policy Optimization (GRPO) has emerged as a leading paradigm for optimizing models on self-generated trajectories. However, the on-policy nature of GRPO bounds the model to the reasoning skills it can already produce, restricting to learn more advanced capabilities. Prior works inject privileged reasoning traces from a stronger teacher policy to guide training, yet these traces are inherently out of distribution with respect to the student policy. We observe that this mismatch between on-policy and off-policy causes gradient clipping on semantically critical reasoning tokens, ultimately rewarding correct answers while leaving the reasoning that justifies them unlearned. Hence, we propose \textbf{Echo-GRPO}, a framework that lets the model reason in the words it speaks. Rather than imitating low-probability privileged traces from the teacher model, Echo-GRPO rewrites them into the student policy's own \textit{idiolect}, that is, its own characteristic vocabulary and expression patterns, while preserving their semantics via Dual-Reference Decoding. We instantiate this framework as \textbf{VideoEcho-R1} for video reasoning distillation, achieving consistent improvements across three multimodal LLM backbones and five benchmarks. Finally, we show that our idiolectal paraphrasing is a plug-in module that consistently improves both RL and supervised fine-tuning frameworks for reasoning distillation, demonstrating that policy-aligned supervision extends beyond GRPO.
