Search papers, labs, and topics across Lattice.
This paper introduces Flow Continuous Trajectory Supervision (FlowCTS), a novel method for enhancing flow models by leveraging on-policy distillation to mitigate issues of sparse rewards and exposure bias. By matching student and reference trajectories from the same initial state and deriving a temporally weighted velocity-matching upper bound, FlowCTS-OPD significantly improves performance metrics such as GenEval and OCR compared to traditional KL-based OPD and mixed-reward RL baselines. The findings reveal that FlowCTS-OPD not only accelerates convergence but also addresses temporal supervision mismatches inherent in existing methods, demonstrating its effectiveness in both on-policy and supervised fine-tuning settings.
FlowCTS-OPD boosts performance metrics by over 3% while solving the temporal supervision mismatch that plagues traditional methods.
While on-policy distillation (OPD) effectively addresses sparse rewards and exposure bias in large language model post-training, its extension to flow models remains underexplored. To this end, we propose Flow Continuous Trajectory Supervision (FlowCTS), which matches subsequent student and reference trajectories initialized from the same student-visited state. Using the integral relation between trajectories and velocity fields, we derive a temporally weighted velocity-matching upper bound and discretize it into practical objectives parameterized by the number of supervision steps. Under a multi-reference setup, single-state FlowCTS-OPD outperforms vanilla KL-based OPD with faster convergence. FlowCTS-OPD improves GenEval from 0.90 to 0.93, OCR from 0.90 to 0.92, and PickScore from 22.75 to 23.06, while outperforming a mixed-reward RL baseline across all target metrics. Further analysis reveals a clear temporal supervision mismatch in vanilla KL-based OPD arising from its auxiliary SDE transition kernels. Beyond on-policy setting,FlowCTS also consistently outperforms vanilla SFT , particularly on OCR, while increasing supervision steps exhibit a trade-off between richer trajectory information and greater optimization difficulty.
