Search papers, labs, and topics across Lattice.
This paper introduces a video-rate streaming pipeline for stylization using a vision-aware multimodal large language model (MLLM) conditioned edit diffusion, leveraging an aggressively distilled U-Net to alleviate the per-frame bottleneck in real-time text-to-image generation. By implementing asymmetric side-stream and main-stream CUDA pipelining, a fused graph formulation, and a periodic conditioning-refresh schedule, the authors achieve impressive throughput rates of 27.4 fps on an RTX 3090 Ti and up to 74.1 fps on an RTX 5090. The approach not only demonstrates significant efficiency improvements but also showcases strong generalization capabilities across various unseen video sequences and style families.
Achieving video-rate stylization at 27.4 fps on consumer hardware, this pipeline redefines the efficiency of real-time text-to-image generation.
Aggressive distillation of the diffusion U-Net inverts the per-frame bottleneck of real-time text-to-image pipelines: once the denoiser is a 4-step or 1-step distilled student, the text encoder becomes the critical path. This inversion is most acute in vision-aware edit diffusion, where the encoder is a multimodal large language model (MLLM). We study the case of a 0.39B distilled edit U-Net paired with a 2.13B MLLM text encoder (Qwen3-VL) and present a streaming pipeline targeted at this regime built around three engineering mechanisms: asymmetric side-stream / main-stream CUDA pipelining with batched text-encoder amortisation (and optional static-prompt caching), a compile-friendly ControlNet-LLLite reformulation that folds the entire U-Net + adapter stack into a single fused graph, and a periodic conditioning-refresh schedule with a hook subset that amortises the per-frame conditioning cost. On a single consumer RTX 3090 Ti at 512x512 the pipeline sustains 27.4 fps over a 480-frame run at batch size B=8 and 29.6 fps at B=16, with end-to-end p50 latency of approximately 0.5 and 1.0 seconds respectively; the same operating point measures 54.9 fps on RTX 4090 and 74.1 fps on RTX 5090. We report video-rate streaming throughput rather than interactive low latency, and locate our numbers against same-stack StreamDiffusion re-runs as systems context, not as a benchmark superiority claim. For the trained oil-painting style, the released temporal adapter generalises within in-clip noise to 19 unused DAVIS-2017 sequences and 15 non-DAVIS clips from seven sources; prompt-level generalisation to unseen style families is bounded and reported separately.
