Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
StreamErase achieves sub-second latency for video object removal by progressively transferring erasing capabilities from a bidirectional foundation model to an efficient unidirectional streaming eraser. This is accomplished through a novel sliding-window causal attention mechanism with attention sinks and KV cache rolling, which simulates long-horizon inference within a fixed context window to prevent error accumulation and maintain visual fidelity. The resulting model enables real-time, infinite-length streaming video erasing, outperforming existing methods on the newly introduced LongErase benchmark.
Real-time video object removal is now possible at 26 FPS, enabling interactive editing of arbitrarily long videos without latency or quality degradation.
Current video object removal methods suffer from prohibitive latency (minutes per video) and non-causal processing that prevents real-time interaction. We present StreamErase, enabling sub-second latency with up to 26 FPS streaming generation on a single GPU. Our approach first augments a text-to-video model with mask input to generate high-quality videos that erase target objects and synthesize coherent backgrounds, but does not perform inference on the fly. As such, we progressively transfer the erasing capability from a powerful bidirectional foundation model to an efficient unidirectional streaming eraser, enabling long-horizon erasing without sacrificing visual fidelity. Several key challenges arise when generating videos over long, potentially infinite horizons: (1) bridging the domain gap from training on finite length and extrapolating to long horizons, (2) sustaining high quality by preventing error accumulation, and (3) maintaining fast inference, without increasing computational cost as the context grows. To address these challenges, we carefully designed sliding-window causal attention with attention sinks. By incorporating self-rollout with attention sinks and KV cache rolling during training, we simulate inference-time extrapolations with a fixed context window, enabling constant-speed generation of arbitrarily long videos. Finally, we establish a dedicated benchmark, LongErase, for streaming long-video object removal. Our model achieves strong performance compared to existing video object erasing models and generalizes well to real-world video scenarios, uniquely enabling infinite-length streaming. With StreamErase, users can input a mask and see the erasing results unfold in real-time, delivering a truly interactive experience.