Search papers, labs, and topics across Lattice.
MixTok addresses the trade-off between representation compactness and fine-grained detail in visual tokenization by introducing a hybrid approach combining a global 1D representation with sparse, content-adaptive 2D refinement tokens. This method dynamically allocates 2D tokens to high-priority regions identified by a router, replacing corresponding features from the 1D representation to reconstruct the image. MixTok achieves state-of-the-art reconstruction and generation performance on ImageNet 256x256 with significantly reduced token budgets.
Image reconstruction and generation quality rivaling dense tokenizers are achieved with ~130 tokens by adaptively refining global 1D representations with sparse 2D tokens.
Efficient visual tokenization requires balancing representation compactness with the preservation of fine-grained image content. Conventional grid-based tokenizers allocate tokens uniformly across spatial locations despite non-uniform visual complexity, whereas compact 1D tokenizers aggregate global information but can struggle to preserve local details. We introduce MixTok, a hybrid visual tokenizer that combines a compact 1D global representation with sparse 2D tokens for local refinement. A content-adaptive router ranks candidate locations by refinement priority and determines the number of 2D tokens for each image under a target average token budget. At the highest-scoring locations, 2D token features replace the spatial features recovered from the 1D representation, and a shared decoder reconstructs the image from the resulting mixed features. On ImageNet 256 x 256, MixTok achieves highly competitive reconstruction and generation performance, with an rFID of 1.195 and a gFID of 1.95 at respective average budgets of approximately 128 and 131 content tokens per image.
