Search papers, labs, and topics across Lattice.
This paper introduces MultiToP, a framework designed to reduce hallucinations in Video Large Multimodal Models by refining unreliable visual tokens prior to language generation. The method employs a Visual Token Patcher that predicts token-level replacement distributions and utilizes information-guided rank calibration to enhance token replacement based on answer-conditioned cues. Experimental results show that MultiToP significantly decreases hallucinations, improving F1 scores by 50.60% while maintaining overall video understanding performance.
Hallucinations in video models can be mitigated by intelligently patching visual tokens, leading to a 50.60% improvement in response accuracy.
Video Large Multimodal Models have achieved remarkable progress in video understanding, yet they remain prone to hallucinations, where generated responses are not faithfully supported by the input video. In this paper, we propose MultiToP, a multimodal-context-aware visual token patching framework that mitigates hallucinations by refining unreliable visual tokens before language generation. MultiToP introduces a lightweight Visual Token Patcher to predict token-level replacement distributions and selectively substitute unreliable visual tokens with a dynamic global patch token. To train the patcher effectively, we further propose information-guided rank calibration, which uses answer-conditioned frame-level information cues derived from the backbone to guide token replacement. Combined with ground-truth answer supervision and sparsity regularization, MultiToP enables localized visual evidence refinement without modifying the original model. Extensive experiments demonstrate that MultiToP effectively reduces hallucinations on Vript-HAL with negligible inference overhead, improving the F1 scores of Qwen3-VL-4B-Instruct by 50.60% over the vanilla model. Meanwhile, MultiToP preserves general video understanding ability, yielding an 18.58% relative accuracy gain on ActivityNet-QA for Video-LLaVA-7B.
