Search papers, labs, and topics across Lattice.
This paper introduces Self-OPD, a novel teacher-free on-policy distillation framework for flow matching models that utilizes the model's own self-exploration as a source of supervision. By generating multiple stochastic candidates and optimizing the velocity field with a pull-push objective based on normalized advantages, Self-OPD effectively mitigates the issues of high computational costs and error compounding associated with traditional teacher-student paradigms. Experimental results demonstrate that Self-OPD surpasses existing reinforcement learning and on-policy distillation methods, achieving superior performance without the need for task-specific teachers.
Self-OPD achieves state-of-the-art performance in flow matching models by eliminating the need for a separate teacher, transforming self-exploration into effective supervision.
On-policy distillation (OPD), which leverages a pre-trained, specialized teacher model to provide dense supervisory signals, has achieved significant success in Large Language Models (LLMs) and has recently been adapted to flow matching models. However, this paradigm suffers from two major issues: First, training a separate, task-specific teacher for every new objective incurs high computational costs. Second, the discrepancy between teacher and student distributions often leads to compounding errors along the generation trajectory. In this paper, we introduce \textbf{Self-OPD}, a teacher-free OPD framework for flow matching models that turns the student's own self-exploration into step-wise supervision. At each timestep, Self-OPD branches the deterministic next-state prediction into $K$ stochastic SDE candidates, rolls them out with the ODE sampler, and compares their rewards against a deterministic self-reference baseline to obtain normalized advantages. The velocity field is optimized with an all-branch pull-push objective, where high-advantage branches attract the student and low-advantage branches repel it under direction-aware attenuation and SDE-variance normalization. For multi-objective alignment, Self-OPD fuses normalized scores at the reward level, avoiding direct gradient conflict. Experiments on single and mixed reward benchmarks show that Self-OPD outperforms prior RL and OPD methods without task-specific teachers.
