Search papers, labs, and topics across Lattice.
This paper addresses the problem of spatial relationship hallucinations in foreground-conditioned image inpainting, where generated backgrounds exhibit implausible relationships with the given foreground. The authors introduce InpaintDPO, a Direct Preference Optimization (DPO) framework tailored for spatial rationality in this task, comprising MaskDPO to focus optimization on the background, Conditional Asymmetric Preference Optimization to improve boundary coherence, and Shared Commonality Preference Optimization to learn spatial commonalities from high-quality samples. Experiments demonstrate that InpaintDPO effectively mitigates spatial relationship hallucinations and improves the overall quality of foreground-conditioned inpainting.
Spatial relationship hallucinations in image inpainting can be significantly reduced by directly optimizing for preferences on background plausibility, even when foregrounds are identical.
Foreground-conditioned inpainting, which aims at generating a harmonious background for a given foreground subject based on the text prompt, is an important subfield in controllable image generation. A common challenge in current methods, however, is the occurrence of Spatial Relationship Hallucinations between the foreground subject and the generated background, including inappropriate scale, positional relationships, and viewpoints. Critically, the subjective nature of spatial rationality makes it challenging to quantify, hindering the use of traditional reward-based RLHF methods. To address this issue, we propose InpaintDPO, the first Direct Preference Optimization (DPO) based framework dedicated to spatial rationality in foreground-conditioned inpainting, ensuring plausible spatial relationships between foreground and background elements. To resolve the gradient conflicts in standard DPO caused by identical foreground in win-lose pairs, we propose MaskDPO, which confines preference optimization exclusively to the background to enhance background spatial relationships, while retaining the inpainting loss in the foreground region for robust foreground preservation. To enhance coherence at the foreground-background boundary, we propose Conditional Asymmetric Preference Optimization, which samples pairs with differentiated cropping operations and applies global preference optimization to promote contextual awareness and enhance boundary coherence. Finally, based on the observation that winning samples share a commonality in plausible spatial relationships, we propose Shared Commonality Preference Optimization to enhance the model's understanding of spatial commonality across high-quality winning samples, further promoting shared spatial rationality.
