Search papers, labs, and topics across Lattice.
This paper systematically evaluates the impact of component-wise quantization on small vision-language models (VLMs) for on-device deployment, focusing on the Jetson Orin NX and AGX platforms. The findings reveal that quantization sensitivity is more influenced by the structural paradigm of the model, such as mixture of experts (MoE) versus dense architectures, rather than model size alone, with MoE architectures showing resilience to INT4 noise. Additionally, the study highlights that while INT4 quantization reduces VRAM usage, it introduces dequantization overhead that slows token generation, emphasizing the need for hardware-aware deployment strategies.
MoE architectures outperform dense models in quantization resilience, revealing that structure trumps size in on-device VLM performance.
The emergence of vision language models with fewer than 3 billion parameters has accelerated the implementation of on-device multimodal intelligence. However, a detailed understanding of component-wise quantization remains a bottleneck for optimal deployment. This paper presents a systematic evaluation framework for empirically validating five hypotheses across six quantization configurations on the Jetson Orin NX and AGX. By separating the vision encoder, projector, and large language model backbone yields the following results: (1) Quantization sensitivity is governed by the structural paradigm (MoE vs. dense) rather than scale alone, with MoE backbones mitigating INT4 noise where dense backbones degrade; (2) SigLIP encoders incur disproportionate INT8 latency on Jetson Ampere--a deployment-specific encoder-kernel-hardware interaction, not a SigLIP flaw; (3) Although INT4 quantization of LLMs greatly reduces VRAM consumption, it also causes slower token generation due to dequantization overhead; (4) Composite quantization errors are largely additive, except along the modality-alignment path, which is architecture-dependent; (5) The intelligence-per-joule profile varies significantly across platforms owing to memory bandwidth constraints.
