Search papers, labs, and topics across Lattice.
This paper introduces ModernMOE (MMOE), a novel architecture for diffusion transformers that integrates efficient expert design principles from large language models (LLMs) to enhance generation quality while managing training and deployment costs. By systematically adapting various expert components, such as routed experts and attention-residual information reuse, MMOE demonstrates improved convergence rates and lower Fr茅chet Inception Distance (FID) scores compared to both dense and sparse-expert baselines. The findings indicate that AFMs can achieve a better quality-cost balance by leveraging established efficiency mechanisms rather than merely increasing parameter counts and sparsity ratios.
MMOE achieves faster convergence and superior generation quality in diffusion transformers by effectively integrating expert routing strategies, challenging the notion that more parameters always lead to better performance.
Modern large language models scale successfully by pairing capacity growth with efficiency, keeping per-token and deployment costs under control as capacity grows. AIGC Foundation Models (AFMs), especially diffusion-transformer backbones, have begun to adopt sparse experts, but recent efforts mostly enlarge total parameter counts and sparsity ratios without importing the efficiency mechanisms that made LLM scaling practical, so generation quality is seldom balanced against training and deployment cost. This raises a natural question: can the architectural principles behind efficient LLM scaling be adapted to AFMs in a more balanced way? We introduce ModernMOE (MMOE), a modernization of SiT-style diffusion transformers that systematically adapts routed experts, shared and lightweight experts, gate-residual routing, and attention-residual information reuse to AIGC generation. Rather than treating MoE as a single plug-in replacement, MMOE studies how different modern expert components affect convergence, efficiency, and generation quality when composed inside a diffusion transformer. Every experiment in this paper is trained on a single eight-GPU H100 node with batch size 256 for 400k steps, an accessible single-machine budget. Under matched training and sampling protocols and at this budget, MMOE reaches lower FID at every recorded checkpoint, that is, it converges faster per training step, than dense and intermediate sparse-expert baselines, and among the sparse variants it attains the best quality-cost balance. Routing analysis further shows stable expert specialization across depth, substantial use of lightweight routes, and modest step-to-step routing changes during denoising. These results suggest that AFMs can follow the balanced scaling path of LLMs by importing proven efficiency designs, rather than by simply increasing total parameters and sparsity ratios.
