Search papers, labs, and topics across Lattice.
This paper introduces Gazer, a training-free framework that enhances autoregressive visual models (AVMs) by incorporating multimodal large language model feedback to correct semantic errors during the generation process. By employing a two-stage approach鈥擱eflective Diagnosis for identifying errors in intermediate states and Semantic Correction for rectifying the generation trajectory鈥擥azer effectively improves semantic alignment and compositional accuracy. Experimental results on various benchmarks reveal that Gazer significantly enhances output quality without the computational burden of additional training.
Gazer achieves semantic correction in autoregressive visual models without any training, leading to improved output quality and accuracy.
Autoregressive visual models (AVMs) based on next-scale prediction have emerged as a prominent paradigm for image and video synthesis. However, decomposing the generation process into discrete scales with varying granularities in AVM makes semantic errors difficult to identify and correct, thereby undermining the quality of the final output. Prior efforts to enhance AVM can be categorized into training-based and training-free approaches. Although training-based efforts to enhance AVM generation quality come at substantial computational cost, existing training-free methods neglect intermediate generation states, leaving semantic errors undiagnosed and allowing them to accumulate into the final output. In this paper, we focus on training-free paradigms and propose Gazer, a framework that integrates multimodal large language model feedback into the AVM sampling loop for in-generation semantic correction. Concretely, Gazer operates via two cooperating stages: the Reflective Diagnosis stage diagnoses semantic errors from intermediate states, while the Semantic Correction stage rewinds and rectifies the generation trajectory to realign with the target prompt. Experiments on compositional image and video benchmarks demonstrate that Gazer improves semantic alignment and compositional accuracy across multiple AVMs without additional training.
