Search papers, labs, and topics across Lattice.
EarlyTom is introduced as a training-free token compression framework that performs early-stage visual token compression within the vision encoder of Video-LLMs. By compressing tokens early in the encoding process and using a decoupled spatial token selection strategy, EarlyTom significantly reduces the time-to-first-token (TTFT) and FLOPs. Experiments on the LLaVA-OneVision-7B model show a 2.65x reduction in TTFT and a 61% reduction in FLOPs on an A100 GPU, while maintaining accuracy.
Video-LLMs can achieve up to 2.65x faster time-to-first-token and 61% FLOPs reduction by compressing visual tokens *inside* the vision encoder, not just after.
Video large language models (Video-LLMs) have demonstrated strong capabilities in video understanding tasks. However, their practical deployment is still hindered by the inefficiency introduced by processing massive amounts of visual tokens. Although recent approaches achieve extremely low token retention ratios while maintaining accuracy comparable to full-token baselines, most of them perform compression only at the late stage of prefilling, leaving the efficiency of the vision encoder unoptimized. In this paper, we first show that vision encoding contributes a large portion to the time-to-first-token (TTFT). Therefore, instead of compressing visual tokens only after the vision encoder, performing compression inside the encoder still leaves substantial room for exploration. Based on this insight, we propose EarlyTom, a training-free token compression framework that performs early-stage visual token compression inside the vision encoder, enabling significantly better TTFT reduction and higher throughput. In addition, we introduce a decoupled spatial token selection strategy that improves the overall compression effectiveness. EarlyTom reduces TTFT by up to 2.65x and FLOPs by up to 61% on a single NVIDIA A100 GPU for the LLaVA-OneVision-7B model, while maintaining accuracy comparable to the full-token baseline. These improvements substantially enhance the practicality of deploying Video-LLMs in real-world production scenarios.
