Search papers, labs, and topics across Lattice.
The paper addresses the problem of insufficient spatial precision in Medical Visual Grounding (MVG) when using Vision-Language Models (VLMs). They introduce KnowMVG, a framework that enhances spatial awareness by incorporating knowledge-enhanced prompts and a global-local attention mechanism. Experiments on four MVG benchmarks show that KnowMVG outperforms existing methods, achieving significant gains in AP50 and mIoU, demonstrating the effectiveness of explicitly strengthening spatial awareness during decoding.
VLMs struggle with precise localization in medical imaging, but injecting medical knowledge and using global-local attention can boost grounding accuracy by up to 3% AP50 and 2.6% mIoU.
Medical Visual Grounding (MVG) aims to identify diagnostically relevant phrases from free-text radiology reports and localize their corresponding regions in medical images, providing interpretable visual evidence to support clinical decision-making. Although recent Vision-Language Models (VLMs) exhibit promising multimodal reasoning ability, their grounding remains insufficient spatial precision, largely due to a lack of explicit localization priors when relying solely on latent embeddings. In this work, we analyze this limitation from an attention perspective and propose KnowMVG, a Knowledge-prior and global-local attention enhancement framework for MVG in VLMs that explicitly strengthens spatial awareness during decoding. Specifically, we present a knowledge-enhanced prompting strategy that encodes phrase related medical knowledge into compact embeddings, together with a global-local attention that jointly leverages coarse global information and refined local cues to guide precise region localization. localization. This design bridges high-level semantic understanding and fine-grained visual perception without introducing extra textual reasoning overhead. Extensive experiments on four MVG benchmarks demonstrate that our KnowMVG consistently outperforms existing approaches, achieving gains of 3.0% in AP50 and 2.6% in mIoU over prior state-of-the-art methods. Qualitative and ablation studies further validate the effectiveness of each component.
