Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper presents a novel approach for conditional inpainting using a frozen text-to-image diffusion model, eliminating the need for inpainting-specific training or dataset adaptation. By employing a method called Step-PI, which integrates boundary-interior latent feedback and a structured release schedule, the authors achieve significant performance improvements across three natural-image domains. The results demonstrate that their method outperforms existing training-free baselines while maintaining the flexibility of a generic model, highlighting its effectiveness in cross-domain applications.
A frozen text-to-image diffusion model can achieve superior inpainting results across multiple domains without any inpainting-specific training, challenging the need for tailored optimization.
We show that a frozen generic text-to-image diffusion model can perform conditional inpainting across three evaluated natural-image domains with one fixed controller configuration, without inpainting-specific weight training, dataset-specific weight adaptation, or learned inpainting-specific conditioning channels. Step-PI augments known-region projection with boundary-interior latent feedback, persistent PI state, and a predefined four-field release schedule that modulates controller signals along the reverse trajectory. Developed only on Main35-disjoint CelebA-HQ pilots, the controller transfers unchanged to AFHQ and Places2. Across two field-identical comparisons on the same 3,500 cases, adding persistent state and replacing uniform release with the predefined schedule each improve all 15 dataset-metric cells; 95% bootstrap intervals exclude zero for all five metrics in both comparisons. In descriptive native-route comparisons, Step-PI leads LanPaint and PILOT (the closest evaluated training-free baselines using vanilla SD1.5) on all five equal-dataset macro metrics. Inpainting-trained systems retain the absolute metric leads but rely on substantial inpainting-specific offline optimization. Our method provides a complementary approach for repurposing a frozen generic text-to-image model for cross-domain inpainting through test-time latent control.