Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces Elastic Looped Transformers (ELT), a recurrent transformer architecture with weight sharing across iterative blocks, for parameter-efficient visual generation. To train these models, they propose Intra-Loop Self Distillation (ILSD), distilling intermediate loop configurations from the maximum loop configuration to ensure consistency. ELT achieves a competitive FID of 2.0 on class-conditional ImageNet 256x256 with a 4x parameter reduction under iso-inference-compute settings.
Achieve state-of-the-art image and video generation with 4x fewer parameters by sharing weights across transformer blocks in a recurrent architecture.
We introduce Elastic Looped Transformers (ELT), a highly parameter-efficient class of visual generative models based on a recurrent transformer architecture. While conventional generative models rely on deep stacks of unique transformer layers, our approach employs iterative, weight-shared transformer blocks to drastically reduce parameter counts while maintaining high synthesis quality. To effectively train these models for image and video generation, we propose the idea of Intra-Loop Self Distillation (ILSD), where student configurations (intermediate loops) are distilled from the teacher configuration (maximum training loops) to ensure consistency across the model's depth in a single training step. Our framework yields a family of elastic models from a single training run, enabling Any-Time inference capability with dynamic trade-offs between computational cost and generation quality, with the same parameter count. ELT significantly shifts the efficiency frontier for visual synthesis. With $4\times$ reduction in parameter count under iso-inference-compute settings, ELT achieves a competitive FID of $2.0$ on class-conditional ImageNet $256 \times 256$ and FVD of $72.8$ on class-conditional UCF-101.