Search papers, labs, and topics across Lattice.
This study investigates the compression of visual-token embeddings in vision-language models (VLMs) by employing a standardized codec, specifically ISO/IEC 15938-17 Neural Network Coding (NNC), to reduce the payload of communicated intermediate tensors. The authors demonstrate that closed-ended Video-MME accuracy remains nearly intact with up to 98% reduction in transmitted BF16 tensor size, revealing a plateau-and-collapse behavior in performance metrics. Notably, the findings suggest that downstream reasoning relies more on the coarse structure and relative geometry of the data rather than precise floating-point values, advocating for a shift towards rate-task optimization in AI traffic codecs.
Achieving a 98% reduction in AI traffic without sacrificing performance challenges conventional wisdom about the necessity of precise data representation in VLMs.
When the visual encoder and the language decoder of a vision-language model (VLM) run on different compute nodes, the intermediate visual-token embeddings become a communicated payload rather than an internal activation. We call such machine-consumed intermediate tensors AI traffic and ask how far they can be compressed with a standardized, training-free codec. We insert ISO/IEC 15938-17 Neural Network Coding (NNC) round trips on the complete visual interface of a Qwen3-VL-8B-Instruct video question answering pipeline, comprising the main visual-token representation and the DeepStack feature streams, while leaving weights, prompts, and generation untouched, and sweep the quantization parameter (QP) over a wide rate range. Closed-ended Video-MME accuracy remains close to the uncompressed reference up to a 98% reduction of the transmitted BF16 tensor and only then collapses; open-ended MLVU generation shows the same plateau-and-collapse profile under an LLM judge. This robustness is not due to near-lossless reconstruction: the decoded tensor is heavily discretized, carries substantial row-wise relative L2 error, and has a visibly steeper singular-value decay than its source. Downstream reasoning therefore depends on coarse structure and relative geometry rather than exact floating-point values, which argues for rate-task rather than rate-distortion optimization of AI traffic codecs.
