Search papers, labs, and topics across Lattice.
This study investigates the primacy bias in multimodal knowledge-based visual question answering (KB-VQA) systems, revealing that the position of retrieved information significantly influences performance. Through a controlled gold-position protocol, the authors demonstrate that information presented at the beginning of the context leads to superior outcomes compared to information at the end, with improvements ranging from 16 to 26 points across various models and benchmarks. The findings suggest that traditional metrics like recall@k may not adequately capture the performance of KB-VQA systems, highlighting the need for reader-side interventions to address this bias.
"Lost at the End" reveals a striking primacy bias in multimodal KB-VQA, where early context dramatically outperforms late context, challenging conventional evaluation metrics.
Knowledge-based visual question answering (KB-VQA) lets vision-language systems answer questions that exceed their parametric knowledge by conditioning a reader on passages retrieved from a Wikipedia-scale knowledge base. In pure-text long-context LLMs, retrieved-context use follows the U-shaped "lost-in-the-middle" effect of Liu et al. (2024): information at the start and end of context is used, the middle is lost. Whether this transfers to deployed multimodal KB-VQA is open. To close this gap, we design the first controlled probe of reader-side position dependence in multimodal KB-VQA: a gold-position protocol in which only the gold passage's prompt slot varies within question. We run it on three open-source 7B/8B VLM readers and two KB-VQA benchmarks at k up to 20. The shape flips from U to primacy: gold-at-first beats gold-at-last by 16 to 26 points on every reader-by-benchmark cell, an effect we call "Lost at the End". Three targeted ablations narrow the cause: a text-only control shows the multimodal setting amplifies an already-present text-mode primacy 2.2 to 4.5 times, and image-position and distractor-shuffle ablations together pin the locus to prompt slot 0 of the instruction-tuned reader. On a frozen reader, three retrieval-side fixes (MMR, oracle reranking, rank-based reordering) all leave the gap intact (no separable improvement). Our findings indicate that recall@k is the wrong metric for deployed KB-VQA and that closing the gap requires reader-side intervention; we release our protocol as a controlled instrument for evaluating such interventions.
