Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces MAR-MAER, a hierarchical autoregressive text-to-image generation framework designed to improve image quality and handle ambiguous prompts. It employs a metric-aware embedding regularization using a lightweight projection head trained with an adaptive kernel regression loss to align internal representations with human-preferred metrics like CLIPScore and HPSv2. Additionally, a conditional variational module is incorporated to introduce controlled randomness, enabling the generation of diverse and coherent images from ambiguous prompts.
Autoregressive image generators can now better align with human aesthetic preferences and generate diverse outputs for ambiguous prompts, thanks to a novel metric-aware regularization and conditional variational module.
Autoregressive (AR) models have demonstrated significant success in the realm of text-to-image generation. However, they usually face two major challenges. Firstly, the generated images may not always meet the quality standards expected by humans. Furthermore, these models face difficulty when dealing with ambiguous prompts that could be interpreted in several valid ways. To address these issues, we introduce MAR-MAER, an innovative hierarchical autoregressive framework. It combines two main components. It is a metric-aware embedding regularization method. The other one is a probabilistic latent model used for handling ambiguous semantics. Our method utilizes a lightweight projection head, which is trained with an adaptive kernel regression loss function. This aligns the model's internal representations with human-preferred quality metrics, such as CLIPScore and HPSv2. As a result, the embedding space that is learned more accurately reflects human judgment. We are also introducing a conditional variational module. This approach incorporates an aspect of controlled randomness within the hierarchical token generation process. This capability allows the model to produce a diverse array of coherent images based on ambiguous or open-ended prompts. We conducted extensive experiments using COCO and a newly developed Ambiguous-Prompt Benchmark. The results show that MAR-MAER achieves excellent performance in both metric consistency and semantic flexibility. It exceeds the baseline Hi-MAR model's performance, showing an improvement of +1.6 in CLIPScore and +5.3 in HPSv2. For unclear inputs, it produces a notably wider range of outputs. These findings have been confirmed through both human evaluation and automated metrics.