Search papers, labs, and topics across Lattice.
This paper introduces AR-CoPO, a reinforcement learning from human feedback (RLHF) framework tailored for streaming autoregressive video generation, addressing the limitations of existing SDE-based GRPO methods. AR-CoPO employs chunk-level alignment through a forking mechanism to generate neighborhood candidates and localized GRPO updates, coupled with a semi-on-policy training strategy that balances exploration and exploitation. Experiments on Self-Forcing demonstrate AR-CoPO's superior out-of-domain generalization and in-domain human preference alignment, suggesting improved genuine alignment.
RLHF for autoregressive video generation gets a boost with AR-CoPO, which overcomes the limitations of SDE-based methods by using chunk-level alignment and a semi-on-policy training strategy.
Streaming autoregressive (AR) video generators combined with few-step distillation achieve low-latency, high-quality synthesis, yet remain difficult to align via reinforcement learning from human feedback (RLHF). Existing SDE-based GRPO methods face challenges in this setting: few-step ODEs and consistency model samplers deviate from standard flow-matching ODEs, and their short, low-stochasticity trajectories are highly sensitive to initialization noise, rendering intermediate SDE exploration ineffective. We propose AR-CoPO (AutoRegressive Contrastive Policy Optimization), a framework that adapts the Neighbor GRPO contrastive perspective to streaming AR generation. AR-CoPO introduces chunk-level alignment via a forking mechanism that constructs neighborhood candidates at a randomly selected chunk, assigns sequence-level rewards, and performs localized GRPO updates. We further propose a semi-on-policy training strategy that complements on-policy exploration with exploitation over a replay buffer of reference rollouts, improving generation quality across domains. Experiments on Self-Forcing demonstrate that AR-CoPO improves both out-of-domain generalization and in-domain human preference alignment over the baseline, providing evidence of genuine alignment rather than reward hacking.
