Search papers, labs, and topics across Lattice.
The paper introduces SPATIALALIGN, a self-improvement framework that uses zeroth-order regularized Direct Preference Optimization (DPO) to fine-tune text-to-video (T2V) models for improved alignment with dynamic spatial relationships (DSR) specified in text prompts. They propose a geometry-based metric, DSR-SCORE, to quantitatively evaluate the alignment between generated videos and DSRs, addressing limitations of VLM-based evaluations. Experiments demonstrate that fine-tuning with SPATIALALIGN significantly improves the baseline T2V model's ability to depict spatial relationships accurately.
Text-to-video models can now generate videos that actually respect spatial relationships, thanks to a new geometry-based reward function that beats VLM-based alternatives.
Most text-to-video (T2V) generators prioritize aesthetic quality, but often ignoring the spatial constraints in the generated videos. In this work, we present SPATIALALIGN, a self-improvement framework that enhances T2V models capabilities to depict Dynamic Spatial Relationships (DSR) specified in text prompts. We present a zeroth-order regularized Direct Preference Optimization (DPO) to fine-tune T2V models towards better alignment with DSR. Specifically, we design DSR-SCORE, a geometry-based metric that quantitatively measures the alignment between generated videos and the specified DSRs in prompts, which is a step forward from prior works that rely on VLM for evaluation. We also conduct a dataset of text-video pairs with diverse DSRs to facilitate the study. Extensive experiments demonstrate that our fine-tuned model significantly out performs the baseline in spatial relationships. The code will be released in Link.
