Search papers, labs, and topics across Lattice.
AdaSpark is introduced as an adaptive sparsity framework for efficient long-video processing with Video-LLMs, partitioning videos into 3D spatio-temporal cubes and employing context-aware selection. The framework uses Adaptive Cube-Selective Attention (AdaS-Attn) to attend to relevant video cubes and Adaptive Token-Selective FFN (AdaS-FFN) to process salient tokens within each cube, both driven by an entropy-based selection mechanism. Experiments show AdaSpark achieves up to 57% FLOPs reduction while maintaining performance comparable to dense models on hour-scale video benchmarks.
Achieve near-dense Video-LLM performance on long videos with up to 57% fewer FLOPs by adaptively selecting which video cubes and tokens to process.
Processing long-form videos with Video Large Language Models (Video-LLMs) is computationally prohibitive. Current efficiency methods often compromise fine-grained perception through irreversible information disposal or inhibit long-range temporal modeling via rigid, predefined sparse patterns. This paper introduces AdaSpark, an adaptive sparsity framework designed to address these limitations. AdaSpark first partitions video inputs into 3D spatio-temporal cubes. It then employs two co-designed, context-aware components: (1) Adaptive Cube-Selective Attention (AdaS-Attn), which adaptively selects a subset of relevant video cubes to attend for each query token, and (2) Adaptive Token-Selective FFN (AdaS-FFN), which selectively processes only the most salient tokens within each cube. An entropy-based (Top-p) selection mechanism adaptively allocates computational resources based on input complexity. Experiments demonstrate that AdaSpark significantly reduces computational load by up to 57% FLOPs while maintaining comparable performance to dense models and preserving fine-grained, long-range dependencies, as validated on challenging hour-scale video benchmarks.
