Search papers, labs, and topics across Lattice.
The paper introduces SeqLoRA, a bilevel optimization framework for parameter-efficient fine-tuning of text-to-image diffusion models to compose multiple custom concepts. SeqLoRA jointly optimizes LoRA factors with orthogonal regularization to mitigate representation interference and catastrophic forgetting during continual learning of new concepts. Experiments show SeqLoRA improves identity preservation and scalability across a large number of concepts compared to existing methods, while also providing theoretical convergence and forgetting bounds.
Forget about messy concept soups – SeqLoRA lets you teach your diffusion model 100+ new tricks without them blurring together.
Parameter-efficient fine-tuning enables fast personalization of text-to-image diffusion models, but composing multiple custom concepts remains challenging due to representation interference. Existing modular methods either rely on expensive post-hoc fusion or freeze adaptation subspaces, which limit expressiveness and concept fidelity. To address this trade-off, we propose Sequential regularized LoRA (SeqLoRA), a constrained continual learning framework that jointly optimizes both LoRA factors via bilevel optimization. Theoretically, we establish strong convergence guarantees for our algorithm and model the residual layer activations as a matrix sub-Gaussian process to derive high-probability bounds on catastrophic forgetting. We further prove that learning the LoRA basis from data minimizes residual interference energy more effectively than frozen-basis methods. Experiments on multi-concept image generation demonstrate that SeqLoRA improves identity preservation and scalability across up to 101 concepts, while avoiding costly fusion and reducing attribute interference in composed generations.
ETH