Search papers, labs, and topics across Lattice.
This paper introduces MagicPrompt, a novel framework for ultra-lightweight prompt tuning in video generation that addresses the inefficiencies of existing parameter-efficient fine-tuning methods. By employing Attention-Embedded Prompt Tuning, it significantly reduces the number of trainable parameters while maintaining the integrity of pre-trained knowledge, and it enhances reward optimization stability through Dual-Space Reward Feedback Optimization. Experimental results demonstrate that MagicPrompt achieves competitive performance with less than 1% of the trainable parameters typically required, leading to substantial reductions in training costs.
Achieving competitive video generation performance with under 1% of the trainable parameters typically needed could revolutionize how we approach fine-tuning in large-scale models.
Large-scale video diffusion models (VDMs) deliver strong generation performance, but full fine-tuning for downstream tasks incurs prohibitive computational costs. Existing parameter-efficient fine-tuning (PEFT) methods have two critical flaws on billion-scale models: they still require substantial trainable parameters, and reward-based training suffers from noise-induced optimization instability in condition-guided tasks. We propose MagicPrompt, a lightweight framework that achieves extreme parameter efficiency and stable reward optimization. It first adopts Attention-Embedded Prompt Tuning, which steers generation via lightweight soft prompts with orders of magnitude fewer parameters while preserving pre-trained knowledge. It further introduces Dual-Space Reward Feedback Optimization, which uses self-supervised latent objectives to improve condition-guided reward training. Experiments show MagicPrompt reaches competitive performance with less than 1\% trainable parameters and notably reduces training costs.
