Search papers, labs, and topics across Lattice.
SemanTok introduces a novel flexible video tokenizer that leverages frozen DINO features and reconstructs them from token prefixes using lightweight heads, addressing limitations of prior REPA-based tokenizers. This approach significantly improves semantic alignment and video fidelity across autoregressive model sizes, enabling a 201M SemanTok model to outperform a 3.4x larger VideoFlexTok model. The method demonstrates robustness to out-of-distribution classes and enhances decoder performance even at high noise levels, while also reducing prediction costs for short prefixes.
A single video tokenizer can achieve superior semantic alignment and fidelity compared to much larger models by directly conditioning on frozen visual features.
Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse tokens carry the clip's global semantics while later tokens further specify details. Existing flexible tokenizers only apply a representation-alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. SemanTok achieves high semantic alignment and video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model $3.4\times$ its size, and larger SemanTok AR models further improve fidelity. It keeps semantic alignment on out-of-distribution classes and gives the decoder higher semantic alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later tokens.
