Search papers, labs, and topics across Lattice.
The paper introduces HiLo-Token, an input-adaptive token compression framework that optimizes the allocation of computational resources for image editing tasks in Diffusion Transformers (DiTs). By prioritizing high-frequency regions for token allocation and utilizing downsampled images for low-frequency areas, the method significantly reduces latency while maintaining image quality. Experimental results demonstrate that HiLo-Token achieves speedups of 3.13x, 2.59x, and 1.67x for varying mask ratios, addressing a critical bottleneck in generative AI image editing tools.
Achieving up to 3.13x speed improvements in image editing tasks without sacrificing quality could revolutionize the efficiency of generative AI tools.
Creative image editing tools, such as Photoshop's Remove or Generative Fill buttons, are central to everyday customer use and account for a major share of traffic in Photoshop and Lightroom. However, current generative AI models face significant latency challenges, which become even more pronounced when transitioning from convolution-based U-Nets to Diffusion Transformers (DiTs). In our evaluation on hundreds of representative image editing samples spanning a wide range of mask ratios, the DiT module alone accounts for an average of 73% of the total model latency, even after being distilled from 50 timesteps down to 8 timesteps. To tackle this challenge, we propose $\textbf{HiLo-Token}$, an input-adaptive token compression framework that allocates more token budget to high-frequency, rich-context regions while assigning fewer tokens to low-frequency areas. Specifically, for the editing region specified by the user mask, we retain all tokens within a dilated mask to preserve strong locality and contextual relevance. Outside the editing region, we introduce a simple yet effective high-frequency token selection strategy based on spatial frequency to capture important local details, while using tokens from a 16x downsampled image to represent low-frequency components and preserve the blurry but global structure. Extensive experiments on production-level evaluation data validate the effectiveness of the proposed method, achieving 3.13x, 2.59x, and 1.67x DiT speedups on A100-80GB for image editing tasks across small, medium, and large mask ratio categories with average ratios of 6.38%, 15.92%, and 35.36%, respectively, without any regression in generation quality.
