Search papers, labs, and topics across Lattice.
This paper introduces BeyondMasks, a novel benchmark for evaluating video object removal that emphasizes causal and physical consistency rather than merely masked region fidelity. By providing temporally aligned synthetic and real-world video pairs with clean background references, the dataset allows for a more comprehensive assessment of object removal's impact on the scene's physical effects. The authors also propose the CORE evaluation protocol, which reveals that state-of-the-art methods often fail to address secondary physical effects, highlighting a critical gap between visual realism and causal accuracy in current approaches.
State-of-the-art video object removal methods achieve high visual fidelity but systematically fail to maintain causal consistency in real scenes.
Recent advances in generative video models have significantly improved visual realism in video object removal, yet evaluation protocols still focus on masked region fidelity, treating removal as local inpainting. In real scenes, object removal is a causal intervention: eliminating an object also requires removing its induced physical effects, such as shadows, reflections, illumination changes, translucency, and dynamic traces. Existing benchmarks lack aligned clean references or remain limited to simplified synthetic settings, preventing systematic evaluation of causal consistency. We introduce BeyondMasks, a paired benchmark for causally consistent video object removal, consisting of temporally aligned synthetic and real world video pairs with clean background references. The dataset spans diverse photometric, geometric, volumetric, and dynamic interactions, and supports both mask based and instruction driven editing. We further propose CORE, a structured vision language model based evaluation protocol that jointly measures object disappearance and after effect consistency, aligning more closely with human judgments than existing metrics. Benchmarking state of the art methods reveals systematic failures in removing secondary physical effects despite high masked region fidelity, exposing a gap between visual plausibility and causal correctness. BeyondMasks reframes video object removal as causal scene consistency rather than local reconstruction and provides a unified framework for its evaluation.
