Search papers, labs, and topics across Lattice.
This paper introduces UniHEAR, a novel framework designed to enhance Knowledge-Based Visual Question Answering (KB-VQA) by addressing the limitations of single-source retrieval and retrieval-source-blind reranking. By implementing a Coarse Retrieval Descriptor and Retrieval-Guided Attentive Modality Gating, UniHEAR effectively integrates multiple retrieval modalities and improves the attention mechanism based on retrieval origins. Experimental results on E-VQA and InfoSeek show that UniHEAR outperforms existing methods, achieving a 6.7-point increase in Recall@1 while maintaining a lightweight architecture.
UniHEAR boosts retrieval performance in KB-VQA by 6.7 points, overcoming the limitations of single-source retrieval and blind reranking.
Knowledge-Based Visual Question Answering (KB-VQA) requires retrieving relevant entity knowledge from external sources to answer visually grounded questions. Existing retrieval-augmented systems suffer from two critical limitations. First, relying on a single retrieval modality creates a Single-Source Retrieval Bottleneck, missing ground-truth entities that are only accessible through complementary sources. Second, dual-tower pointwise rerankers suffer from Retrieval-Source-Blind Reranking, as they overlook retrieval origins and candidate-level retrieval priors, leading to redundant modality reliance. To address these challenges, we propose UniHEAR, a unified lightweight framework for heterogeneous-source entity retrieval and reranking. UniHEAR constructs a Coarse Retrieval Descriptor for each candidate entity, and introduces Retrieval-Guided Attentive Modality Gating to condition modality attention weights on this descriptor, further complemented by Entropy-Weighted Source Fusion of coarse retrieval priors. A hybrid training strategy combining contrastive learning with an auxiliary modality-preserving loss unifies entity-level and section-level retrieval within a single model. Extensive experiments on E-VQA and InfoSeek demonstrate that UniHEAR achieves state-of-the-art retrieval and VQA performance, improving Recall@1 by 6.7 and 1.2 points over the strongest baselines while maintaining a lightweight reranking architecture. Code and model are available at https://github.com/iven-luo/UniHEAR.
