Track genomics, neuroscience and molecular biology.
Search papers, labs, and topics across Lattice.
The paper introduces TrajTok, an end-to-end learnable video tokenizer that generates trajectory-based tokens by implicitly clustering pixels in space and time, thereby decoupling token count from video duration. This approach avoids reliance on external segmentation and tracking pipelines, enabling efficient and adaptable tokenization co-trained with downstream video models. Experiments demonstrate that TrajTok improves video understanding performance, achieving state-of-the-art accuracy on classification and retrieval benchmarks with a video CLIP model (TrajViT2), and proving versatile as a probing head (TrajAdapter) or alignment connector (TrajVLM).
Ditch slow, external segmentation pipelines: TrajTok learns trajectory tokens end-to-end, boosting video understanding while staying lean and adaptable.
Tokenization in video models, typically through patchification, generates an excessive and redundant number of tokens. This severely limits video efficiency and scalability. While recent trajectory-based tokenizers offer a promising solution by decoupling video duration from token count, they rely on complex external segmentation and tracking pipelines that are slow and task-agnostic. We propose TrajTok, an end-to-end video tokenizer module that is fully integrated and co-trained with video models for a downstream objective, dynamically adapting its token granularity to semantic complexity, independent of video duration. TrajTok contains a unified segmenter that performs implicit clustering over pixels in both space and time to directly produce object trajectories in a single forward pass. By prioritizing downstream adaptability over pixel-perfect segmentation fidelity, TrajTok is lightweight and efficient, yet empirically improves video understanding performance. With TrajTok, we implement a video CLIP model trained from scratch (TrajViT2). It achieves the best accuracy at scale across both classification and retrieval benchmarks, while maintaining efficiency comparable to the best token-merging methods. TrajTok also proves to be a versatile component beyond its role as a tokenizer. We show that it can be seamlessly integrated as either a probing head for pretrained visual features (TrajAdapter) or an alignment connector in vision-language models (TrajVLM) with especially strong performance in long-video reasoning.
Apple ML