Search papers, labs, and topics across Lattice.
This study investigates the relationship between the number of visual tokens and end-to-end latency in multimodal inference, revealing that fewer tokens do not necessarily lead to faster processing times. By employing a stage-level decomposition approach, the authors reconcile decision overhead and shared work with measured latency, highlighting the limitations of autoregressive probes compared to full models. Notably, a pre-vision image-size rule demonstrates significant potential for reducing latency by avoiding preprocessing and vision encoding, outperforming post-vision strategies in specific contexts.
Reducing visual tokens doesn't always mean faster inference; a pre-vision strategy can significantly cut latency by bypassing preprocessing steps.
Fewer visual tokens do not guarantee lower end-to-end latency. We evaluate break-even with a reproducible protocol that accounts for decision overhead, shared work, and the operators each policy can avoid. A stage-level decomposition reconciles these components with measured end-to-end latency. In a 30-example pilot, the two tested autoregressive probes remain slower than Full despite state reuse. A lightweight post-vision predictor yields paired confidence intervals below zero on RTX 3090 and A100 and remains significant after a conservative all-pairs Holm correction. A pre-vision image-size rule also yields intervals below zero on both GPUs, although neither comparison remains significant after the same correction. Pre-vision routing has a structural opportunity unavailable to post-vision pruning: it can avoid preprocessing and vision encoding. On A100, this opportunity outweighs a nearly eightfold larger downstream token reduction by the post-vision policy. Reported quality is conditional on examples answered correctly by Full and is not benchmark accuracy.
