Search papers, labs, and topics across Lattice.
The paper introduces ORBIS, a hardware-software co-designed accelerator for video Diffusion Transformers (DiT) that reduces computational cost by improving token reduction. ORBIS uses output activations from the previous timestep to enhance inter-token similarity estimation, leading to higher matching quality and token reduction ratios. They further introduce a Distribution-Aware Token Matching (DATM) algorithm and specialized hardware to minimize latency and hardware cost.
Video diffusion models can be accelerated by 4.5x with a novel hardware-software co-design that leverages output activations to guide token reduction, achieving significantly higher compression ratios than prior art.
Diffusion Transformer (DiT) has emerged as a powerful model architecture for generating high-quality images and videos. In the case of video DiT, 3D Spatio-Temporal Attention increases token length in proportion to the number of frames, sharply increasing computational cost. Token reduction methods mitigate this cost by exploiting spatial redundancy, but existing approaches rely on inaccurate similarity estimates and lightweight matching algorithms, resulting in poor matching quality and only marginal acceleration. To overcome these limitations, we propose ORBIS, an SW-HW co-designed accelerator for video DiT. ORBIS leverages the output activation from the previous timestep to obtain more accurate inter-token similarity, substantially improving matching quality and enabling a higher token reduction ratio. We further introduce a Distribution-Aware Token Matching (DATM) algorithm that captures global token distribution and explicitly minimizes token-pair loss for additional gains. To fully hide DATM latency, we design specialized, deeply pipelined hardware and minimize its hardware cost through quantization, occupying only 2.4% of total area with negligible accuracy loss. Extensive experiments show that ORBIS achieves about 2x higher token reduction ratio than the state-of-the-art approach, AsymRnR, while delivering up to 4.5x speedup and 79.3% energy reduction compared to an NVIDIA A100 GPU.
