Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to adaptive video tokenization that leverages temporal redundancy in the latent space of a frozen continuous video tokenizer, allowing for efficient token allocation without the need for computationally expensive decoder passes. By applying a parameter-free mechanism that identifies and discards redundant latent positions based on temporal-L1 differences, the method dynamically adjusts token budgets according to the visual complexity of the sequence. The proposed Latent Inpainting Transformer (LIT) effectively reconstructs dropped positions while achieving a significant speedup in inference time, outperforming existing continuous and discrete adaptive tokenization methods.
Exploiting temporal redundancy in video tokenization can yield a staggering 31x speedup in inference time while maintaining high reconstruction fidelity.
Adaptive video tokenisation seeks to dynamically allocate token budgets based on the underlying visual complexity of a sequence. Current continuous-regime approaches achieve this via iterative binarised searches or trained neural regressors, while discrete methods often require a full-rate decoder pass to estimate information content. We demonstrate that such computational overheads are not strictly necessary. We show that the latent space of a frozen continuous video tokeniser inherently encodes temporal redundancy that can be exploited directly: spatial positions whose latent representations change minimally between consecutive frames carry near-zero additional information. We introduce a parameter-free adaptive token allocation mechanism that applies a fixed threshold to per-position temporal-L1 differences, identifying and dropping redundant latent positions. Consequently, the compression rate emerges naturally from the input content rather than being enforced top-down: static scenes get compressed aggressively, while highly dynamic sequences retain more tokens. To reconstruct the dropped positions, we propose the Latent Inpainting Transformer (LIT), a lightweight factorised spatial-temporal attention architecture. The resulting inference pipeline is highly efficient, requiring only a single encoder pass and one LIT forward pass, eliminating the need for auxiliary routing networks. Evaluations across TokenBench and DAVIS, which are the standard benchmarks used by recent tokenisers~\cite{infotok, agarwal2025cosmos}, indicate that our framework yields meaningful, content-driven token allocation while maintaining competitive reconstruction fidelity, and delivers a $31\times$ inference-time speedup over the continuous adaptive baseline (ElasticTok-CV) and an $\approx2\times$ speedup over the discrete information-theoretic baseline (InfoTok)
