Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Multi-Token Localized Attention (MTLA), a training-free method that enhances the grounding confidence of multimodal large language models (MLLMs) by measuring the attention of prediction tokens to their claimed regions. The authors demonstrate that MTLA significantly improves the ability to discern accurate localized predictions, outperforming previous attention-based detectors by focusing attention summation within the claimed regions. Results show that MTLA boosts hallucination AUROC by 7 to 38 points across various MLLM families and modalities, and it nearly doubles the zero-shot COCO detection AP of an 8B generalist model, effectively bridging the performance gap to supervised detectors.
MTLA transforms how we assess grounding confidence in MLLMs, achieving up to 38% better performance in reducing hallucinations without any additional training.
Multimodal large language models can emit localized predictions, bounding boxes for objects and temporal windows for video and audio events, but they hallucinate these regions prolifically. The model's own token log-probabilities are nearly uninformative: they conflate grounding quality with input ambiguity, and coordinate tokens become near-deterministic once the model commits. We propose Multi-Token Localized Attention (MTLA): a training-free, post-hoc score that measures how strongly a prediction's tokens attend to the region they claim. Prior attention-based detectors, which sum attention over the entire input modality and read a single response token, are weaker special cases; we show that summing only within the claimed region and aggregating across all prediction tokens recovers a stronger grounding signal. The same recipe applies almost trivially to other modalities and tasks: object detection in images and temporal localization in video and audio. Across multiple MLLM families and three modalities, MTLA improves hallucination AUROC by +7 to +38 over the best prior training-free baseline. Used as a confidence score for re-ranking, it nearly doubles the zero-shot COCO detection AP of an open-source 8B generalist (from 20.4 to 37.0), narrowing the gap to supervised detectors without any task-specific training.