Search papers, labs, and topics across Lattice.
This paper introduces ReToken, a novel learnable embedding that enhances vision-language models by efficiently selecting relevant visual tokens from a large context. By addressing the computational limitations of processing extensive visual data, ReToken demonstrates significant performance improvements on various benchmarks, including a 13.4-point gain on Visual Haystacks with Qwen3VL-8B. The lightweight design allows for effective training and inference on a single H100 GPU, making it a practical solution for long video retrieval tasks.
ReToken boosts vision-language model performance by over 20% on key benchmarks while fitting training and inference within a single GPU's memory constraints.
Long visual context poses a challenge for vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once is computationally infeasible under GPU memory constraints. We present ReToken, a single learnable embedding trained as an explicit retrieval target that selects a sparse set of query-relevant visual tokens from a pre-filled visual KV cache. Trained on only a small image-QA dataset, ReToken yields consistent gains across image and video benchmarks: on Visual Haystacks it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (>20% relative), and on LVBench it transfers zero-shot to long video for an 8.0-point gain with Qwen3VL-8B. Thanks to its lightweight design, both training and long-video inference fit on a single H100. Code is available at: https://github.com/avaxiao/ReToken
Microsoft Research