Search papers, labs, and topics across Lattice.
The paper introduces U-turn chains, a Markov chain sampling method that iterates short forward-backward steps of a diffusion model with Metropolis-Hastings correction to sample from energy-modified targets while remaining on the learned data manifold. Through experiments on synthetic and natural data, the authors demonstrate that minimal U-turn dynamics can exhibit an ergodicity-breaking phase transition due to fragmentation of the data manifold, leading to slow relaxation, particularly for high-level features. They observe an inversion in the relaxation ordering of low-level and high-level features only at larger U-turn magnitudes, suggesting strongly constrained local dynamics.
Diffusion models can get stuck in local modes, with high-level semantic features relaxing much slower than low-level details, hindering efficient sampling.
Sampling from learned high-dimensional distributions is a foundational computational problem. We introduce U-turn chains: Markov chains obtained by iterating short forward-backward steps of a diffusion model, in which each step proposes a move that remains on the learned data manifold and, paired with a Metropolis-Hastings correction, samples from energy-modified targets. For synthetic languages, we show that minimal U-turn dynamics undergoes an ergodicity-breaking phase transition driven by fragmentation of the data manifold; ergodicity is restored at larger U-turn magnitude. In the non-ergodic regime, low-level features relax faster than high-level ones, an ordering that inverts only at sufficiently large U-turn magnitude. We test these predictions on natural language and natural images. In both modalities, minimal U-turns relax slowly, especially for high-level features approximated by deep representations in CNNs or LLMs. The layer-ordering inversion appears only at large noise when mixing is efficient -- signatures consistent with strongly constrained, weakly mixing local dynamics. We discuss the implications of these results for sampling with diffusion models.
