Search papers, labs, and topics across Lattice.
This paper introduces a novel intrinsic reward mechanism for training multi-modal large language models (MLLMs) to use zoom-in tools effectively, eliminating the need for extensive supervised fine-tuning. By employing an InfoNCE-style reward with a contrastive curriculum of increasingly difficult negative tool calls, the authors demonstrate that their method achieves competitive performance on benchmark datasets while significantly improving efficiency. Notably, their approach outperforms all baselines when used as a drop-in replacement for traditional supervised fine-tuning, highlighting the effectiveness of their training paradigm.
Forget costly supervised fine-tuning鈥攖his new contrastive reward mechanism enables efficient zoom-in tool learning in MLLMs, outperforming traditional methods.
Using a zoom-in tool is an important foundational part of modern visual agents, because it allows to efficiently handle tasks involving high-resolution images. Most previous methods need an extensive warm-start supervised fine-tuning phase for teaching models zoom-in. We show that this is not necessary by proposing a new intrinsic reward for learning tool use in MLLMs without the need for additional labels or warm-start SFT. Our InfoNCE-style reward uses a curriculum of increasingly hard negative tool calls as a contrastive training signal. Empirical experiments on $V^*$, HRBench and MME-RealWorld show that our approach is competitive while being more efficient. When used as a drop-in replacement for SFT, we even outperform all baselines. To directly measure the zoom-in ability of models, we further introduce the scalable synthetic Muffin&Chihuahua (M&C) dataset. Each image consists of a grid with every cell either showing a muffin or chihuahua. Leveraging the M&C dataset's unique region of interest labels, we find that recall is the metric that most strongly correlates the zoom-in region with final task performance. Our model and code for reproduction is publicly available under https://github.com/UKPLab/emnlp2026-zoom-in
