Search papers, labs, and topics across Lattice.
This paper addresses the issue of measurement-conditioned hallucination in diffusion-based inverse problem solvers, where generated content may appear realistic but lacks consistency with the actual measurements. By analyzing the separation of prior updates and measurement conditioning in Bayes-rule-based diffusion solvers, the authors introduce the Robust Prior Update (RPU) module, which stabilizes the prior update process without altering the measurement correction. The results show that RPU significantly enhances both automatic metrics and human preferences in image reconstruction tasks on FFHQ and ImageNet datasets, indicating improved instance faithfulness in the generated outputs.
Robustifying the prior update in diffusion models can drastically enhance the fidelity of image reconstructions, achieving over 91% preference in human evaluations.
Diffusion-based inverse problem solvers can produce realistic reconstructions, but realism alone does not ensure that the recovered details are supported by the measurement. We study this failure as measurement-conditioned hallucination: visually meaningful content that is either implausible or inconsistent with the measured instance. Our analysis separates Bayes-rule-based diffusion inverse solvers into a prior update and a measurement-conditioning step, showing that hallucinated content can enter through the prior-side proposal before the measurement correction is applied. Motivated by this view, we propose Robust Prior Update (RPU), a solver-level module that probes the local stability of the diffusion prior update, re-anchors the resulting displacement at the current iterate, and leaves the measurement update unchanged. We instantiate RPU in DPS and evaluate it on FFHQ and ImageNet inverse problems using automatic metrics and human faithfulness studies. On FFHQ, RPU improves PSNR and LPIPS over DPS across box inpainting, Gaussian deblurring, and motion deblurring. In human judgments, RPU receives 91.9% of blind non-tie majority preferences and 91.1% of ground-truth-assisted non-tie preferences on FFHQ box inpainting, while the ImageNet Gaussian reader study is tie-heavy but favors RPU among non-tie cases. These results support a targeted claim: robustifying the prior update can improve instance faithfulness in diffusion inverse solvers, especially when the prior shapes weakly constrained content.
