Track new results across pure and applied mathematics.
Search papers, labs, and topics across Lattice.
This paper investigates stylistic idiosyncrasies in captioning models and their propagation into text-to-image generation by training classifiers to predict the originating caption model from either generated captions or corresponding images. The study reveals that caption models exhibit highly distinguishable stylistic signatures detectable with near-perfect accuracy via text classification, while these signatures are significantly diminished in generated images. Further analysis identifies a failure to preserve key variations from captions in the generated images as the cause of this cross-modal discrepancy.
Text-to-image models wash away the unique stylistic fingerprints of their captioning counterparts, revealing a surprising disconnect between text and image generation.
In this work, we study idiosyncrasies in the caption models and their downstream impact on text-to-image models. We design a systematic analysis: given either a generated caption or the corresponding image, we train neural networks to predict the originating caption model. Our results show that text classification yields very high accuracy (99.70\%), indicating that captioning models embed distinctive stylistic signatures. In contrast, these signatures largely disappear in the generated images, with classification accuracy dropping to at most 50\% even for the state-of-the-art Flux model. To better understand this cross-modal discrepancy, we further analyze the data and find that the generated images fail to preserve key variations present in captions, such as differences in the level of detail, emphasis on color and texture, and the distribution of objects within a scene. Overall, our classification-based framework provides a novel methodology for quantifying both the stylistic idiosyncrasies of caption models and the prompt-following ability of text-to-image systems.
TU Delft