Search papers, labs, and topics across Lattice.
This paper introduces UltraViT, a vision encoder specifically designed for large vision-language models (LVLMs) that optimizes for on-device latency, addressing the computational challenges that hinder deployment on resource-constrained devices. By employing a pyramidal architecture with heterogeneous spatial mixers and a novel two-stage generative pre-training strategy, UltraViT achieves significant improvements in semantic grounding and multimodal alignment. Experimental results show that UltraViT operates nearly 1.7 times faster than existing encoder-centric baselines while maintaining high performance, setting a new standard for efficient LVLM encoding.
UltraViT achieves a groundbreaking 1.7x speed increase for on-device vision-language model encoding without sacrificing performance.
Large Vision-Language Models (LVLMs) remain bottlenecked by massive computational footprints, precluding their deployment on resource-constrained edge devices. While efforts to compress LVLMs focus heavily on vision token reduction or smaller language models, the vision encoder is largely overlooked, typically deployed as a monolithic, computationally heavy feature extractor. Moreover, there is no previous effort that designs a vision encoder for LVLMs directly optimized for on-device latency. In this paper, we present UltraViT, a vision encoder for LVLMs, explicitly designed and optimized for on-device performance. Specifically, by taking into account real on-device latencies, we systematically design a pyramidal architecture that strategically integrates and adapts heterogeneous spatial mixers at the macro-block level. Furthermore, to pre-train UltraViT, we propose a novel two-stage generative pre-training strategy: cultivating rich spatial features via dense distillation, followed by direct generative supervision from a capacity-mixed frozen LLM. Compared to standard contrastive and SSL, we show that our pre-training is much more effective for achieving high-level semantic grounding for UltraViT needed for the subsequent generative multimodal alignment of LVLM training. Extensive experiments demonstrate that our on-device latency-informed design combined with our tailored training strategy establishes a new state-of-the-art for efficient LVLM encoding, significantly outperforming existing encoder-centric baselines while operating on-device at nearly 1.7xthe speed.
