Search papers, labs, and topics across Lattice.
This paper introduces VisLens, a novel method for fine-grained visual search in multimodal large language models (MLLMs) that addresses the limitations of existing approaches by combining interpretability and speed. By utilizing a logit lens to decode semantics from hidden states and employing a lightweight tuned-lens for early hidden state mapping, VisLens achieves accurate visual token extraction in a single forward pass. The method significantly outperforms previous baselines, achieving up to 22.2 times faster performance compared to traditional multi-pass search methods while maintaining or exceeding accuracy levels.
VisLens achieves a remarkable speedup in visual search for MLLMs, completing the task in a single pass while maintaining high accuracy.
Multimodal large language models (MLLMs) struggle with fine-grained Visual Search, the task of locating small or rare objects in high-resolution images. Existing remedies fall into two families: (1) Training-free methods based on attention or confidence scores are accurate but slow, since they require multiple MLLM queries per example. (2) Reinforcement Learning (RL) trained tool-use models are faster at inference but opaque, since their tool calls remain uncontrollable and hard to interpret. To overcome this, we propose \emph{VisLens} (Visual Focus via Logit Lens), a Visual Search method built on the logit lens, which decodes the semantics held in a hidden state by projecting it through the LLM head. VisLens further uses a lightweight tuned-lens that maps early hidden states into the final hidden state space, so visual tokens can be read out from early layers. These tokens are matched to target words in the query to generate a crop of the relevant region, which is fed back in alongside the original image to produce the final answer. The whole process, from decoding to the final answer, completes in a single forward pass without repeated queries. VisLens matches or exceeds prior baselines while delivering a substantial latency advantage, running $8.5$--$9.9\times$ faster than Thyme and up to $22.2\times$ faster than training-free multi-pass search methods.
