Search papers, labs, and topics across Lattice.
This paper introduces the Proactive Routing Paradigm (PRP), which enhances inference efficiency in multimodal visual reasoning by enabling early decision-making through a dual evaluation of model competencies. By employing Draft Rating Learning (DRL) and Joint Rating Learning (JRL), the approach allows for adaptive routing of queries to either a smaller draft model or a larger target model based on their respective abilities to handle specific tasks. Experimental results across various multimodal reasoning benchmarks demonstrate that PRP significantly accelerates inference while maintaining high performance levels.
Early decision-making in multimodal reasoning can cut inference time without sacrificing accuracy, thanks to a novel dual evaluation of model competencies.
Large multimodal models have achieved strong reasoning on complex visual tasks, but their inference efficiency is often restricted by long chains of thought. A promising solution is to pair a small draft model with a large target model, enabling cooperative inference employing a routing signal that adaptively routes queries to either the draft or target model based on their difficulties for optimal efficiency and accuracy. Yet, the remaining bottleneck is to establish a reliable query difficulty signal under multimodal settings. Existing approaches designed for language models either rely on post-hoc token probabilities, which fall short in multimodal scenarios, or depend on supervised fine-tuning, which is a data-sensitive strategy. Both paradigms perform routing only after a complete output, and ignore whether the target model can actually solve the routed instances. To address this, we propose PRP, a Proactive Routing Paradigm that enables early decision-making by jointly evaluating the competence of both the draft and target models. Our Draft Rating Learning (DRL) equips the draft model with an internal confidence estimator, while Joint Rating Learning (JRL) predicts how well the target model can handle a given query, thereby prioritizing the allocation of samples it excels at rather than the hardest ones. These ratings enable fine-grained, instance-level \textbf{Proactive Routing} and substantially accelerate inference without compromising overall performance. Extensive experiments across multiple multimodal reasoning benchmarks validate our effectiveness and efficiency.
