Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Masked Logit Nudging (MLN), a novel approach for prompt-guided image editing in visual autoregressive models that preserves unrelated regions of the source image. MLN leverages source image token maps to guide the model's predictions under the target prompt, using cross-attention differences to define spatial masks for editing. Experiments on the PIE benchmark demonstrate that MLN achieves state-of-the-art image editing performance at high resolutions, while also delivering faithful reconstructions and outperforming previous methods on COCO and OpenImages datasets.
Autoregressive models can now edit images with unprecedented fidelity and speed, rivaling diffusion models without sacrificing reconstruction quality.
We address the problem of prompt-guided image editing in visual autoregressive models. Given a source image and a target text prompt, we aim to modify the source image according to the target prompt, while preserving all regions which are unrelated to the requested edit. To this end, we present Masked Logit Nudging, which uses the source image token maps to introduce a guidance step that aligns the model's predictions under the target prompt with these source token maps. Specifically, we convert the fixed source encodings into logits using the VAR encoding, nudging the model's predicted logits towards the targets along a semantic trajectory defined by the source-target prompts. Edits are applied only within spatial masks obtained through a dedicated masking scheme that leverages cross-attention differences between the source and edited prompts. Then, we introduce a refinement to correct quantization errors and improve reconstruction quality. Our approach achieves the best image editing performance on the PIE benchmark at 512px and 1024px resolutions. Beyond editing, our method delivers faithful reconstructions and outperforms previous methods on COCO at 512px and OpenImages at 1024px. Overall, our method outperforms VAR-related approaches and achieves comparable or even better performance than diffusion models, while being much faster. Code is available at'https://github.com/AmirMaEl/MLN'.