Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MMSearch-OPD, a novel framework for training long-horizon multimodal search agents by addressing limitations in challenging data and process feedback. The Refine pipeline generates complex reasoning chains from simpler multimodal questions through parameterized transformations, enabling controllable expansion of search depth and cross-modal evidence requirements. Joint reinforcement learning with on-policy distillation, guided by answer rewards and behavior penalties, effectively trains agents, demonstrating significant improvements in long-horizon multimodal deep search capabilities.
Targeted data refinement and joint RL/OPD training unlock significantly stronger long-horizon multimodal deep search capabilities by making hard problems learnable.
Multimodal deep search is moving from short queries to long-horizon investigations that continually acquire evidence. However, training such agents remains constrained by limited challenging training questions and insufficient process feedback. We introduce MMSearch-OPD, combining controllable data refinement with joint reinforcement learning (RL) and on-policy distillation (OPD). Our Refine pipeline performs targeted expansion of reasoning chains derived from existing simple multimodal questions. Parameterized transformations control search depth and cross-modal evidence requirements while reusing existing image--question pairs. For training, answer rewards and lightweight behavior penalties guide RL, while the reward gap modulates token-level teacher guidance on imperfect student trajectories. An exponentially decaying distillation coefficient reduces teacher dependence as training proceeds. Experiments show gains from data refinement and further improvements from joint training after SFT, indicating stronger long-horizon multimodal deep search capabilities.
Tsinghua AI