Search papers, labs, and topics across Lattice.
This paper introduces Sparse Context, a novel method for enhancing the efficiency of reference-based diffusion models by employing token dropping to create sparse reference representations. By fine-tuning the model with random token dropping and then applying task-aware selection strategies during inference, the authors achieve a significant increase in generation speed—4x for multi-reference and 2x for single reference generation—while maintaining visual quality. This work addresses the computational challenges of reference-based models and demonstrates that substantial portions of reference tokens can be discarded without degrading output fidelity.
Achieving a 4x speedup in multi-reference image generation without sacrificing visual quality by intelligently dropping reference tokens.
Reference-based diffusion models enable highly controllable image generation by leveraging elements from input images to guide prompt-driven synthesis. However, these models are computationally expensive in runtime, and their cost scales severely with the number of input references. While the efficiency of diffusion models has been extensively studied in the context of prompt-driven generation, it remains largely under-explored in the realm of reference-based models. This setting presents unique challenges not addressed by methods focusing solely on generation. In particular, the wasteful representation of references as dense token grids offers significant opportunities for improvement. In this work, we present Sparse Context, a method for constructing sparse reference representations by retaining only a reduced subset of reference tokens. We observe that even without modifying the model, dropping a significant portion of reference tokens at inference time largely preserves its generation capabilities. To fully realize this potential, we fine-tune the model with random token dropping at varying ratios, encouraging robustness to partial reference representations. Crucially, this training strategy decouples the model from any specific token selection rule, allowing flexible control at inference time. At inference time, instead of random dropping, we apply task-aware token selection strategies that prioritize the most informative regions of the reference images, adapting the token budget to the input and task requirements. Extensive experiments show our method achieves a 4x increase in inference speed for multi-reference generation and an 2x for single reference generation. Importantly, this efficiency is achieved without compromising visual quality across both spatially-aligned editing and subject-driven generation.
