Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel self-supervised representation learning method for generative modeling tokenization, achieving state-of-the-art FID-50K of 2.18 with JiT-L by factorizing pointwise mutual information via information differences of conditional predictions. Unlike DINO and LeJEPA, this generative approach avoids dimensional collapse and feature suppression, demonstrating predictable improvements with increased compute and superior FLOP efficiency for pixel-space generation. The method achieves significant gains in representation learning, requiring substantially fewer FLOPs than existing methods on ImageNet and GPIC.
Generative representation learning unlocks compute-optimal latent tokenization, achieving superior image quality with drastically fewer FLOPs than prior methods.
We introduce an approach for self-supervised representation learning designed to be used as a tokenizer for generative modeling. Our objective supervises a representation loss by factorizing the pointwise mutual information of the data, which we estimate by taking information differences between conditional predictions from a learned generative model. We justify our algorithm theoretically and empirically, demonstrating that our generative approach to representation learning avoids dimensional collapse and feature suppression in cases where commonly used methods like DINO and LeJEPA fail. Moreover, we show our representation learning technique is scalable; expending more compute leads to predictably better latent tokenization. For the first time, we perform a careful analysis of the combined flop efficiency of tokenizer and generative model and demonstrate that our approach is a compute-optimal improvement over direct diffusion for pixel-space generation, obtaining an FID-50K of 2.18 with JiT-L. Applied to ImageNet and the text-to-image dataset GPIC, we show tokenizer encoder scaling laws for diffusability, where extrapolating our approach achieves representations in 7x fewer FLOPs than DINOv2 on ImageNet and 8x fewer FLOPs than WebSSL on GPIC.
Stanford HAI