Search papers, labs, and topics across Lattice.
This paper introduces FlashDecoder, a novel pure-Transformer video decoder designed for real-time latent-to-pixel streaming, addressing the limitations of traditional 3D convolutional decoders in terms of speed and memory efficiency. By utilizing a rolling KV cache and a fixed-size temporal window, FlashDecoder maintains fast decoding and low memory usage while ensuring temporal causality without explicit attention masks. The results show that FlashDecoder achieves comparable reconstruction quality to convolutional decoders at 1080p, while being 3.6x-4.7x faster and using up to 11x less memory on a single H100 GPU.
FlashDecoder decodes video frames 3.6x-4.7x faster than traditional methods while maintaining high-quality reconstruction at 1080p.
Real-time video generation demands fast decoding as much as fast denoising, yet current latent video diffusion models rely on 3D convolutional decoders that are slow and memory-intensive at high resolutions or for long video. We introduce FlashDecoder, a fast, memory-efficient pure-Transformer video decoder that decodes latents to pixels frame by frame. At each step, the current frame attends only to a fixed-size window of past frames through a rolling KV cache. The fixed temporal window keeps decoding fast and memory bounded regardless of video length, enabling constant-latency streaming. Because frames are processed sequentially, temporal causality is enforced without explicit attention masks, enabling training at resolutions up to 1080p and matching the reconstruction quality of convolutional decoders. On the Wan2.1 and Wan2.2 latent spaces, FlashDecoder matches each convolutional decoder in reconstruction quality (e.g., 41.55dB vs. 41.49dB PSNR at 1080p) while decoding 3.6x-4.7x faster with up to 11x less memory on a single H100 GPU. With architecture-aware inference optimizations, the speedup widens to 12x.
