Search papers, labs, and topics across Lattice.
This paper introduces Mixture-of-Thought-Tokens (Motto), a novel framework that integrates perception and reasoning in multimodal large language models (MLLMs) for free-form grounding tasks. By employing Spatially-Grounded Thought Tokenization, Motto aligns special tokens with spatial locations, enhancing both visual interpretability and spatial correspondence. Extensive experiments reveal that Motto outperforms existing methods, achieving state-of-the-art results across a variety of grounding tasks, thereby addressing the critical gap between perception and reasoning in MLLMs.
MLLMs can now seamlessly unify perception and reasoning, achieving state-of-the-art performance in free-form multimodal grounding tasks.
Multimodal Large Language Models have made great progress in grounding tasks, yet existing methods still struggle to unify precise localization and complex reasoning. For one thing, text-based methods rely on coordinates or index prediction, severely limiting the perceptual capabilities of the model for dense visual objects. Meanwhile, latent token-based methods employ special tokens without inherent spatial references and use a decoding mechanism that lacks thinking steps, weakening high-level reasoning capabilities. Consequently, developing a unified framework that excels in both perception and reasoning remains challenging. To address this, we propose Mixture-of-Thought-Tokens (Motto), a new free-form multimodal grounding method that bridges the perception-reasoning gap, enabling MLLMs to empower diverse, arbitrary grounding queries. Specifically, we introduce Spatially-Grounded Thought Tokenization to explicitly align special tokens with spatial locations for clear spatial correspondence and visual interpretability. We further design a Context-Adaptive Chain-of-Tokens that dynamically switch grounding modes within an interleaved reasoning chain, achieving robust grounding across tasks of varying complexity. In addition, we construct PR-Bench, a new referring expression comprehension benchmark to evaluate the perception-reasoning gap. Extensive experiments demonstrate that Motto achieves state-of-the-art performance across diverse free-form grounding tasks.
