Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Varest, a novel image-token compression method that leverages semantic fungibility by employing variable-resolution tokenization guided by object and text localizers, followed by in-LLM relevance pruning. Varest achieves significant computational reduction by summarizing semantically coherent regions coarsely while preserving fine detail at object boundaries and text. The method demonstrates superior performance across multiple backbones and benchmarks, retaining a high percentage of full-token performance with a drastically reduced image-token count.
Image tokens can be compressed by 94% by prioritizing semantic detail over spatial uniformity, unlocking efficient vision-language models.
Image-token compression must reduce computation without hurting the semantics. We introduce Varest, guided by semantic fungibility: tokens within a coherent semantic region can be summarized coarsely, while object boundaries and literal text require finer distinctions. Object and text localizers guide variable-resolution merging, preserving spatial coverage and finer detail at boundaries and text. Learned [REL] registers then prune visual tokens after early decoder processing. Training across budgets enables one checkpoint per backbone. Across three backbones and 13 benchmarks, Varest outperforms 15 baselines in mean performance retention at every evaluated budget, retaining 86.10-92.33% of full-token performance with only 5.56% of the original layer-averaged image-token count.