Track new results across pure and applied mathematics.
Search papers, labs, and topics across Lattice.
This paper introduces LILIE, an invertible image encoder trained with a vision-language model to recover captions from noisy features, enabling a language-informed loss for visual generation. By augmenting pixel-space loss with this richer signal, the authors achieve a significant reduction in gFD from 66.84 to 53.01 on the GPIC dataset compared to a pixel-only baseline. This demonstrates the efficacy of language-informed supervision in improving end-to-end visual generation quality.
Language supervision slashes visual generation error by 14 points, outperforming pixel-only losses.
End-to-end visual generation with pixel-space loss has recently achieved competitive performance without the need for learned image tokenizers. In this work, we study whether language-informed supervision can provide a richer training signal than pixel loss for visual generation. We introduce LILIE (pronounced"lily"), a Language-Informed and Lossless Image Encoder. LILIE is an invertible image transform trained such that a vision-language model can recover captions from noised features. The encoder uses a normalizing flow transformer trained jointly with a vision-language model via next token prediction. We then use the encoder to extract a simple language-informed loss for text-to-image generation. We train JiT models with LILIE loss, reducing gFD by 14 points (66.84 $\to$ 53.01) against a pixel-only baseline on the GPIC dataset. Our code, encoders, and generative models are available at https://suchiragarwal.com/lilie.
Stanford HAI