Search papers, labs, and topics across Lattice.
This paper introduces the Beyond the Eye (BEE) paradigm, which enhances multimodal large language models (MLLMs) by integrating self-regulated implicit visual tools into the training process, thereby reducing reliance on external tool calls and image re-encoding. By employing a two-stage training approach that includes formalized chain-of-thought supervised fine-tuning and a self-regulated reward mechanism, BEE encourages models to optimize tool usage based on their internal knowledge and context. The results demonstrate that BEE not only achieves state-of-the-art performance in fine-grained visual perception tasks but also significantly improves inference efficiency compared to existing methods.
BEE reduces inference latency and tool redundancy, achieving state-of-the-art performance in visual perception while maintaining competitive reasoning capabilities.
Recent multimodal large language models (MLLMs) have made remarkable progress on fine-grained perception tasks under the "Thinking with Images" (TwI) paradigm by iteratively performing various visual tool operations. However, this paradigm relies heavily on frequent external tool calls and repeated image re-encoding, which leads to substantial computational overhead and inference latency. To address these issues, we propose Beyond the Eye (BEE), a novel implicit visual tool paradigm centered on self-regulated capability. BEE directly incorporates visual tool invocation behaviors into the training objective and encourages the model to develop a self-regulated invocation mechanism. This design enables the model to adaptively balance internal knowledge and implicit tools, avoiding redundant tool usage while substantially reducing inference latency. Specifically, BEE involves a two-stage training process: (1) Formalized Chain-of-Thought (CoT) Supervised Fine-tuning (SFT). We construct CoT trajectories with structured tool slots and mixed invocation states. This stage activates the model's implicit tool representations and adaptive switching capability. (2) Self-regulated Reward-Driven Alignment. To address redundant tool usage caused by ambiguous cognitive boundaries, we first introduce the Net Tool Gain (NTG) metric to quantify this phenomenon. Based on this observation, we further propose a self-regulated reward mechanism. This mechanism penalizes ineffective tool dependency and encourages the model to perform knowledge routing, ensuring that implicit tools are invoked only when the model's internal knowledge is insufficient. BEE achieves state-of-the-art performance in fine-grained visual perception while remaining competitive in general reasoning tasks and achieving substantial gains in inference efficiency.
