Search papers, labs, and topics across Lattice.
This paper introduces YOSE, a fine-tuning framework for efficient DiT-based video object removal that addresses the high inference latency of existing methods. YOSE employs Batch Variable-length Indexing (BVI) to adaptively select essential tokens based on mask information and a Diffusion Process Simulator (DiffSim) to maintain semantic consistency for masked tokens by approximating the influence of unmasked regions. Experiments show YOSE achieves up to 2.5X speedup in 70% of cases while maintaining visual quality comparable to the baseline, demonstrating mask-aware acceleration.
Achieve up to 2.5X faster video object removal by focusing DiT computations only on the essential tokens dictated by the mask.
Recent advances in Diffusion Transformer (DiT)-based video generation technologies have shown impressive results for video object removal. However, these methods still suffer from substantial inference latency. For instance, although MiniMax Remover achieves state-of-the-art visual quality, it operates at only around 10FPS, primarily due to dense computations over the entire spatiotemporal token space, even when only a small masked region actually requires processing. In this paper, we present YOSE, You Only Select Essential Tokens, an efficient fine-tuning framework. YOSE introduces two key components: Batch Variable-length Indexing (BVI) and Diffusion Process Simulator (DiffSim) Module. BVI is a differentiable dynamic indexing operator that adaptively selects essential tokens based on mask information, enabling variable-length token processing across samples. DiffSim provides a diffusion process approximation mechanism for unmasked tokens, which simulates the influence of unmasked regions within DiT self-attention to maintain semantic consistency for masked tokens. With these designs, YOSE achieves mask-aware acceleration, where the inference time scales approximately linearly with the masked regions, in contrast to full-token diffusion methods whose computation remains constant regardless of the mask size. Extensive experiments demonstrate that YOSE achieves up to 2.5X speedup in 70% of cases while maintaining visual quality comparable to the baseline. Code is available at: https://github.com/Wucy0519/YOSE-CVPR26.
