Search papers, labs, and topics across Lattice.
This paper introduces LILAC, a novel framework for personalizing text-to-image diffusion models by composing independently trained low-rank adapters at inference time, thus avoiding parameter-level interference. By conditioning each subject on a frozen composite of previously placed subjects and activating only one adapter at a time, LILAC effectively preserves individual identities while ensuring spatial and visual coherence in the generated images. The method demonstrates significant improvements in identity preservation, achieving an ArcFace detection rate of 0.861 compared to 0.745 with traditional Orthogonal Adaptation methods.
Composing concepts as separate image layers in LILAC eliminates identity confusion in multi-concept diffusion models, achieving superior results without joint retraining.
Personalizing text-to-image diffusion models to render several specific subjects in a coherent image remains challenging: the model must preserve each subject's identity while keeping the scene spatially and visually coherent. Methods that fuse independently trained concept adapters in a shared weight space (via federated averaging, gradient fusion, or orthogonality constraints) suffer from identity confusion and style bleeding and require joint retraining. In this work, we show that composing concepts as separate image layers, instead of merging their adapters in a shared weight space, avoids parameter-level interference. We introduce LILAC, a framework that composes independently trained low-rank adapters at inference time: each subject is conditioned on the frozen composite of previously placed subjects, with exactly one adapter active at a time, therefore identities never interfere at the parameter level. LILAC composes the adapters without any joint training, scales linearly with the number of concepts, and is backbone-agnostic. Under the Orthogonal Adaptation protocol, LILAC applied on Qwen-Image-Edit reaches an ArcFace detection rate of 0.861, while Orthogonal Adaptation reports 0.745 in its original setting. Adaptation reports 0.745 in its original setting. Code is available at https://github.com/marianlupascu/LILAC.
