Search papers, labs, and topics across Lattice.
The paper introduces the Physics-Aware Video Instance Removal (PVIR) benchmark to evaluate video instance removal methods based on their ability to maintain physical consistency after object removal, going beyond just visual plausibility. PVIR comprises 95 videos with instance masks and removal prompts, split into Simple and Hard subsets focusing on complex physical interactions like shadows and reflections. Evaluation of four methods (PISCO-Removal, UniVideo, DiffuEraser, CoCoCo) using a decoupled human evaluation protocol reveals that while PISCO-Removal and UniVideo perform best, all methods struggle with the Hard subset, indicating limitations in handling complex physical side effects.
Current video editing methods still struggle to maintain physical realism when removing objects, often leaving behind telltale signs like lingering shadows that betray the edit.
Video Instance Removal (VIR) requires removing target objects while maintaining background integrity and physical consistency, such as specular reflections and illumination interactions. Despite advancements in text-guided editing, current benchmarks primarily assess visual plausibility, often overlooking the physical causalities, such as lingering shadows, triggered by object removal. We introduce the Physics-Aware Video Instance Removal (PVIR) benchmark, featuring 95 high-quality videos annotated with instance-accurate masks and removal prompts. PVIR is partitioned into Simple and Hard subsets, the latter explicitly targeting complex physical interactions. We evaluate four representative methods, PISCO-Removal, UniVideo, DiffuEraser, and CoCoCo, using a decoupled human evaluation protocol across three dimensions to isolate semantic, visual, and spatial failures: instruction following, rendering quality, and edit exclusivity. Our results show that PISCO-Removal and UniVideo achieve state-of-the-art performance, while DiffuEraser frequently introduces blurring artifacts and CoCoCo struggles significantly with instruction following. The persistent performance drop on the Hard subset highlights the ongoing challenge of recovering complex physical side effects.
