Search papers, labs, and topics across Lattice.
This paper introduces Style Nursing, a method to improve zero-shot style transfer in text-to-image diffusion models, particularly for complex traffic scenes, by addressing the "Style Neglection" problem. Style Nursing directs the model's attention to style subject tokens in the prompt and excites corresponding visual activations. They also introduce Spatial and Semantic Guidance to preserve content during editing and a new benchmark dataset, STREET-6K, for evaluating traffic scene style transfer.
Style Nursing overcomes "Style Neglection" in diffusion models, enabling faithful zero-shot style transfer in complex scenes like traffic environments, and even boosts downstream perception tasks when used for data augmentation.
Recent advances in text-to-image diffusion models have shown an outstanding ability in zero-shot style transfer. However, existing methods often struggle to balance preserving the semantic content of the input image and faithfully transferring the target style in line with the edit prompt. Especially when applied to complex traffic scenes with diverse objects, layouts, and stylistic variations, current diffusion models tend to exhibit Style Neglection, i.e., failing to generate the required style in the prompt. To address this issue, we propose Style Nursing, which directs the model to focus on style subject tokens in the text prompt and excites their corresponding visual activations. Moreover, we introduce Spatial and Semantic Guidance to guide the preservation of content after editing, which utilizes spatial features from the DDIM sampling process together with attention maps from the semantic reconstruction. To evaluate the performance of zero-shot style transfer methods in traffic scenes, we present STREET-6K, a new benchmark dataset comprising 6000 images showcasing diverse traffic scenes and style transfer variations, accompanied by comprehensive annotations and evaluation metrics. Our approach beats state-of-the-art image translation methods in comprehensive quantitative metrics and human evaluations on traffic scene image synthesis while seamlessly generalizing to various other types of images without training or fine-tuning. Further experiments on detection and segmentation tasks show that fine-tuning perception models on our synthesized images improves Recall and mean Intersection over Union (mIoU) by over 10% and 3% respectively in rarely-seen traffic scenes.
