Search papers, labs, and topics across Lattice.
Video2LoRA is introduced as a framework for semantic-controlled video generation conditioned on a reference video, addressing the challenge of semantic alignment across diverse generation conditions. It uses a hypernetwork to predict personalized LoRA weights for each semantic input, creating adaptive LoRA modules within a frozen diffusion backbone. This approach achieves semantically aligned video generation while preserving style and content variations, without requiring per-condition training, and results in a compact model size of under 150MB.
Forget training separate models for each video generation condition – Video2LoRA lets you control video semantics with a single, lightweight (150MB) model.
Achieving semantic alignment across diverse video generation conditions remains a significant challenge. Methods that rely on explicit structural guidance often enforce rigid spatial constraints that limit semantic flexibility, whereas models tailored for individual control types lack interoperability and adaptability. These design bottlenecks hinder progress toward flexible and efficient semantic video generation. To address this, we propose Video2LoRA, a scalable and generalizable framework for semantic-controlled video generation that conditions on a reference video. Video2LoRA employs a lightweight hypernetwork to predict personalized LoRA weights for each semantic input, which are combined with auxiliary matrices to form adaptive LoRA modules integrated into a frozen diffusion backbone. This design enables the model to generate videos consistent with the reference semantics while preserving key style and content variations, eliminating the need for any per-condition training. Notably, the final model weights less than 150MB, making it highly efficient for storage and deployment. Video2LoRA achieves coherent, semantically aligned generation across diverse conditions and exhibits strong zero-shot generalization to unseen semantics.
