Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces LILIE, an invertible image encoder trained with a vision-language model to recover captions from noisy features, enabling a language-informed loss for visual generation. By leveraging this richer supervisory signal, JiT models trained with LILIE loss achieved a significant reduction in gFD from 66.84 to 53.01 on the GPIC dataset compared to pixel-only baselines. This work demonstrates the efficacy of language-informed supervision in improving end-to-end visual generation quality.
Language-informed supervision slashes generative image fidelity degradation by 14 points, outperforming pixel-loss baselines.
End-to-end visual generation with pixel-space loss has recently achieved competitive performance without the need for learned image tokenizers. In this work, we study whether language-informed supervision can provide a richer training signal than pixel loss for visual generation. We introduce LILIE (pronounced "lily"), a Language-Informed and Lossless Image Encoder. LILIE is an invertible image transform trained such that a vision-language model can recover captions from noised features. The encoder uses a normalizing flow transformer trained jointly with a vision-language model via next token prediction. We then use the encoder to extract a simple language-informed loss for text-to-image generation. We train JiT models with LILIE loss, reducing gFD by 14 points (66.84 $\to$ 53.01) against a pixel-only baseline on the GPIC dataset. Our code, encoders, and generative models are available at https://suchiragarwal.com/lilie.
Stanford HAI