Search papers, labs, and topics across Lattice.
This paper introduces a training-free coarse-guided visual generation method that leverages the h-transform to constrain the sampling process of pretrained diffusion models. By modifying the transition probability with a drift function, the method steers generation towards the desired fine sample based on a coarse reference. A noise-level-aware schedule is introduced to de-weight the guidance term as approximation errors increase, balancing guidance adherence and synthesis quality.
Achieve high-fidelity, training-free visual generation from coarse inputs by cleverly steering pretrained diffusion models with a noise-adaptive h-transform.
Coarse-guided visual generation, which synthesizes fine visual samples from degraded or low-fidelity coarse references, is essential for various real-world applications. While training-based approaches are effective, they are inherently limited by high training costs and restricted generalization due to paired data collection. Accordingly, recent training-free works propose to leverage pretrained diffusion models and incorporate guidance during the sampling process. However, these training-free methods either require knowing the forward (fine-to-coarse) transformation operator, e.g., bicubic downsampling, or are difficult to balance between guidance and synthetic quality. To address these challenges, we propose a novel guided method by using the h-transform, a tool that can constrain stochastic processes (e.g., sampling process) under desired conditions. Specifically, we modify the transition probability at each sampling timestep by adding to the original differential equation with a drift function, which approximately steers the generation toward the ideal fine sample. To address unavoidable approximation errors, we introduce a noise-level-aware schedule that gradually de-weights the term as the error increases, ensuring both guidance adherence and high-quality synthesis. Extensive experiments across diverse image and video generation tasks demonstrate the effectiveness and generalization of our method.
