Search papers, labs, and topics across Lattice.
The paper introduces CleanStyle, a plug-and-play framework for encoder-based text-to-image style transfer diffusion models that addresses the problem of content leakage from style images. CleanStyle uses Singular Value Decomposition (SVD) to identify and dynamically suppress content-related noise in the tail components of the style embedding with a time-aware exponential schedule (CleanStyleSVD). The method further introduces Style-Specific Classifier-Free Guidance (SS-CFG) to provide targeted negative signals, improving prompt fidelity and stylization quality without retraining.
Content leakage in text-to-image style transfer can be dramatically reduced by simply suppressing the tail components of the style embedding using SVD and a time-aware schedule, without any retraining.
Style transfer in diffusion models enables controllable visual generation by injecting the style of a reference image. However, recent encoder-based methods, while efficient and tuning-free, often suffer from content leakage, where semantic elements from the style image undesirably appear in the output, impairing prompt fidelity and stylistic consistency. In this work, we introduce CleanStyle, a plug-and-play framework that filters out content-related noise from the style embedding without retraining. Motivated by empirical analysis, we observe that such leakage predominantly stems from the tail components of the style embedding, which are isolated via Singular Value Decomposition (SVD). To address this, we propose CleanStyleSVD (CS-SVD), which dynamically suppresses tail components using a time-aware exponential schedule, providing clean, style-preserving conditional embeddings throughout the denoising process. Furthermore, we present Style-Specific Classifier-Free Guidance (SS-CFG), which reuses the suppressed tail components to construct style-aware unconditional inputs. Unlike conventional methods that use generic negative embeddings (e.g., zero vectors), SS-CFG introduces targeted negative signals that reflect style-specific but prompt-irrelevant visual elements. This enables the model to effectively suppress these distracting patterns during generation, thereby improving prompt fidelity and enhancing the overall visual quality of stylized outputs. Our approach is lightweight, interpretable, and can be seamlessly integrated into existing encoder-based diffusion models without retraining. Extensive experiments demonstrate that CleanStyle substantially reduces content leakage, improves stylization quality and improves prompt alignment across a wide range of style references and prompts.
