Search papers, labs, and topics across Lattice.
This paper introduces Adaptive Visual Inference Scaling (AVIS), a novel approach that jointly optimizes Visual Context Scaling (VCS) and Visual Reasoning Scaling (VRS) in Vision-Language Models (VLMs) to enhance inference efficiency. By employing Key Diversity Visual (KDV) pruning to reduce redundant visual tokens and adaptive self-consistency for dynamic reasoning rollouts, AVIS significantly improves the accuracy-compute trade-off compared to existing methods that optimize either axis in isolation. The results demonstrate that AVIS maintains high performance while minimizing computational costs, making it a practical solution for real-world applications of VLMs.
AVIS achieves a superior accuracy-compute balance by dynamically adjusting visual context and reasoning depth, outperforming traditional single-axis optimization methods.
Modern Vision-Language Models (VLMs) benefit from chain-of-thought prompting and test-time scaling, but these gains often come with prohibitive inference cost due to large visual contexts and long decoding chains. We view this cost through two coupled axes: Visual Context Scaling (VCS), which controls how much visual evidence is passed to the language model, and Visual Reasoning Scaling (VRS), which controls how much inference-time reasoning search is performed. Existing methods typically optimize one axis at a time, leaving the joint allocation of compute across these axes underexplored. We introduce Adaptive Visual Inference Scaling (AVIS), a lightweight policy that adapts both VCS and VRS per query. AVIS realizes VCS through Key Diversity Visual (KDV) pruning, a training-free $O(N)$ key-based rule for removing redundant visual tokens before prefilling, and realizes VRS through adaptive self-consistency, using a learned difficulty predictor to select the number of reasoning rollouts. AVIS is deployment-friendly and compatible with shared-prefill inference, where all rollouts reuse a single prefilling pass and KV cache. Across diverse image and video reasoning benchmarks, AVIS improves the accuracy--compute trade-off relative to VCS-only and VRS-only baselines, and remains effective on top of RL post-trained VLMs while keeping compute and latency low.
