Search papers, labs, and topics across Lattice.
This paper introduces PG-MAP, a training-free framework for joint inference-time alignment of text-to-image models that optimizes both conditioning and latent states through a Gibbs-MAP approach. By enabling coordinated updates across modalities, PG-MAP significantly enhances alignment metrics like PickScore and Aesthetic across various diffusion models, while also achieving impressive performance with flow-matching models. The method not only outperforms existing baselines but also reveals insights into the varying importance of conditioning and latent optimization based on prompt types, suggesting potential for further improvements with adaptive strategies.
Achieving up to 91.9% PickScore, PG-MAP redefines inference-time alignment by jointly optimizing conditioning and latent variables, outperforming traditional methods.
Inference-time alignment of pretrained text-to-image models is typically performed along a single control axis, such as classifier-free guidance, attention editing, or reward-based latent perturbations. This limitation prevents modeling joint dependencies between conditioning and latent variables and hinders transfer across generative transports. We propose PG-MAP, a training-free framework that formulates inference-time alignment as a trajectory-level Gibbs-MAP / proximal energy optimization over the conditioning $c$ and latent state $z_t$ via a forward-consistency coupling, optionally guided by a frozen preference reward. This joint formulation enables coordinated updates across modalities while remaining compatible with both diffusion and flow-matching models through transport-specific adaptations. Across diffusion backbones (SD~1.5, SDXL), PG-MAP consistently improves alignment metrics such as PickScore and Aesthetic, and can be effectively combined with tuned classifier-free guidance to achieve the strongest overall performance. On flow-matching models (SD3.5-medium), the framework reduces to a latent-only variant, achieving $\mathbf{91.9\%}$ PickScore and $75.7\%$ HPS win rates against a static baseline, with controlled experiments ruling out noise-related artifacts. Human evaluations further confirm consistent preference over strong baselines, including tuned CFG and compute-matched universal guidance. Finally, an oracle-routing analysis shows that the relative importance of conditioning and latent optimization depends on prompt types, surfacing further headroom that a per-prompt selector could exploit.
