Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for quantizing vision-language models (VLMs) to enable efficient inference on mobile devices, addressing the significant memory and compute constraints typically associated with these models. By utilizing the model itself to generate training data and implementing a unique 2.7-bit-per-parameter quantization format, the authors achieve a substantial reduction in model size while maintaining performance. The Llama 3.2 11B Vision Instruct model is successfully compressed to 3.7 GB with 8-bit activations, demonstrating strong performance on standard visual question answering tasks.
Compressing VLMs to a mere 3.7 GB without sacrificing performance could revolutionize mobile AI applications.
Deploying vision-language models (VLMs) on mobile devices is challenging due to their significant memory and compute requirements. We present a framework for quantizing VLMs for efficient inference on resource-constrained hardware. Our approach combines a quantization pipeline that uses the model itself to generate training data and does not require access to the training setup, with a novel 2.7-bit-per-parameter format supporting efficient execution on Arm CPUs. We validate our approach by compressing the Llama 3.2 11B Vision Instruct model to 3.7 GB with 8-bit activations, preserving strong performance on a set of standard visual question answering tasks.
