Search papers, labs, and topics across Lattice.
This work unifies online reinforcement learning and offline alignment for flow models under an f-divergence framework, deriving FlowCPO as an offline forward-KL objective that leverages both preferred and dispreferred pairs without online rollouts. Theoretically, the authors show that forward-KL can be bounded by a non-negative contrastive flow matching loss under linear interpolation, resolving the training instability of FlowDPO's signed regression loss which is unbounded below. In evaluations at CFG 3.0, FlowCPO improves in-domain alignment over FlowDPO (0.84 GenEval and 0.87 OCR vs. 0.81 and 0.74) and achieves top out-of-domain GenEval scores, despite mixed reward performance against rejection fine-tuning.
Contrastive flow matching can upper-bound forward-KL divergence on fixed preference pairs, resolving the training instability of unbounded DPO surrogates without requiring online rollouts.
Preference alignment for flow and diffusion models now spans online reinforcement learning and offline preference optimization, but the relation between these methods remains unclear. In particular, existing forward-process alignment methods require fresh samples from the current model, while offline methods based on fixed preference pairs rely primarily on positive-only fine-tuning or DPO-style likelihood-ratio surrogates. We organize these approaches through a divergence-based framework and introduce FlowCPO, an offline forward-KL objective that uses both preferred and dispreferred samples without online rollouts. For linear interpolation, we show under explicit regularity conditions that the forward-KL objective is bounded by a contrastive flow matching loss, yielding a tractable surrogate on fixed data. We further show that this loss is nonnegative, whereas the signed regression loss of simplified FlowDPO can be unbounded below. In the in-domain setting, FlowCPO achieves higher mean GenEval and OCR scores than the evaluated baselines, reaching 0.84 and 0.87 versus 0.81 and 0.74 for FlowDPO at CFG 3.0. In the out-of-domain setting, the results are mixed, with the best GenEval result but lower reward scores than RFT on several metrics.
