Search papers, labs, and topics across Lattice.
The paper introduces Amber-Image, a framework for efficiently compressing large-scale Diffusion Transformers (DiT) for text-to-image generation, addressing the high computational costs of models like Qwen-Image. They achieve this by using timestep-sensitive depth pruning, local weight averaging for re-initialization, and layer-wise distillation to create Amber-Image-10B, followed by a hybrid-stream architecture and progressive distillation to derive Amber-Image-6B. The resulting models achieve comparable performance to much larger models on benchmarks like DPG-Bench and LongText-Bench while reducing parameters by 70% and requiring significantly less training time (under 2,000 GPU hours).
You can slash DiT parameters by 70% and still beat larger models on text rendering, thanks to a smart compression framework that avoids training from scratch.
Diffusion Transformer (DiT) architectures have significantly advanced Text-to-Image (T2I) generation but suffer from prohibitive computational costs and deployment barriers. To address these challenges, we propose an efficient compression framework that transforms the 60-layer dual-stream MMDiT-based Qwen-Image into lightweight models without training from scratch. Leveraging this framework, we introduce Amber-Image, a series of streamlined T2I models. We first derive Amber-Image-10B using a timestep-sensitive depth pruning strategy, where retained layers are reinitialized via local weight averaging and optimized through layer-wise distillation and full-parameter fine-tuning. Building on this, we develop Amber-Image-6B by introducing a hybrid-stream architecture that converts deep-layer dual streams into a single stream initialized from the image branch, further refined via progressive distillation and lightweight fine-tuning. Our approach reduces parameters by 70% and eliminates the need for large-scale data engineering. Notably, the entire compression and training pipeline-from the 10B to the 6B variant-requires fewer than 2,000 GPU hours, demonstrating exceptional cost-efficiency compared to training from scratch. Extensive evaluations on benchmarks like DPG-Bench and LongText-Bench show that Amber-Image achieves high-fidelity synthesis and superior text rendering, matching much larger models.
