Search papers, labs, and topics across Lattice.
This paper introduces Shell Local Coordinate Coding (Shell-LCC), an innovative method for text-to-video generation that leverages the inherent structure of high-quality Supervised Fine-Tuning (SFT) data as a reward model. By modeling the data manifold and encouraging video latents to reside on this manifold, the authors achieve significant improvements in video quality, particularly in reducing low-level distortions and enhancing high-frequency details. Experiments show that Shell-LCC outperforms traditional methods by mitigating over-smoothing artifacts and motion blur, leading to more realistic video outputs.
Transforming the data manifold into a nearly cost-free reward model could revolutionize text-to-video generation by drastically improving realism and detail without the need for extensive human annotations.
Recent text-to-video (T2V) diffusion models rely heavily on auxiliary reward signals (e.g., via reward models or DPO) to align generated content with human aesthetics and improve realism. These signals, however, incur substantial computational overhead, require costly human annotations, and often yield limited improvement in fine-grained local details. In this paper, we argue that your data manifold is secretly a reward model. By explicitly modeling the manifold structure of high-quality Supervised Fine-Tuning (SFT) data and encouraging video latents to lie on this manifold, we derive dense, differentiable, and nearly cost-free reward signals that significantly improve video quality, particularly in mitigating low-level distortions. Our modeling builds upon Local Coordinate Coding (LCC), which captures the `skeleton' of the manifold. However, directly applying LCC suffers from mean regression, pulling latents toward the geometric mean and losing high-frequency details. We therefore extend it to Shell Local Coordinate Coding (Shell-LCC), which models the manifold `surface' as an isotropic shell to align with the true high-density region. Experiments demonstrate that our approach improves realism, enhances high-frequency details, reduces over-smoothing artifacts, and alleviates motion blur.
