Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Routing Policy Optimization (RPO), a novel method for aligning diffusion models with reinforcement learning rewards by injecting stochasticity into the attention routing mechanism rather than perturbing the denoising trajectory. RPO enables exploration within the generator's existing sampler by stochastically selecting key blocks based on attention mass, thereby perturbing the velocity of denoising transitions. This approach achieves significant training efficiency gains (nearly 4x) and improved generation quality compared to prior score-function methods on text-to-image and text-to-video tasks.
Diffusion models can learn new objectives by stochastically routing attention within their existing architecture, achieving 4x faster RL training without sacrificing generation quality.
Reinforcement learning can align visual generators with reward signals beyond their pretraining objectives, but existing score-function methods typically introduce stochasticity by perturbing the reverse denoising trajectory, leading to costly rollout-based optimization. We instead ask whether stochastic exploration can be introduced inside the generator while preserving its original sampler. Routing Policy Optimization (RPO) answers this question by turning attention routing into a stochastic exploration action. At a few selected denoising levels, each query block keeps its own block and samples further key blocks in proportion to the generator's native attention mass, or keeps dense attention. The sampled route perturbs the velocity of that transition, and a Gaussian likelihood around the dense velocity scores the perturbation with a score-function objective. RPO further uses a controlled two-rollout estimator: two rollouts share the same prompt, initial latent, routed denoising levels, and dense anchor layers, differing only in their sampled attention routes. Their reward difference provides an advantage signal for the routed transitions. Because the likelihood differentiates the dense velocity itself, the learned updates remain effective after routing is removed. At inference, RPO restores dense attention and retains the original deterministic sampler. Experiments on text-to-image and text-to-video generation show that RPO improves generation quality while achieving nearly 4x higher training efficiency than Flow-GRPO-style methods. Please see our project page at https://velox-world.com/rpo for visual results.
NUS