Search papers, labs, and topics across Lattice.
This paper introduces EVIS, an Event-Aware Video Instructed Segmentation Assistant that decomposes videos into distinct events using learnable Event Queries, allowing for a more nuanced understanding of complex video content. By leveraging text-guided queries, EVIS effectively partitions videos into simple, event-related segments, mitigating confusion and hallucinations common in traditional methods. The proposed Object-Pixel-Hybrid Learning further enhances the model's ability to track targets over long video sequences, leading to significant performance improvements across five public benchmarks.
EVIS achieves a hierarchical understanding of video content by segmenting it into distinct events, drastically reducing confusion in referring video segmentation tasks.
Existing referring video segmentation methods often treat a video as a single event consisting of multiple images, overlooking the fact that a video typically contains multiple distinct events. Under such a mechanism, the model needs to directly understand all the complex content in the video and text, which can easily lead to confusion and hallucinations. To address this issue, we propose to decompose a video to a set of simple events by learnable Event Query, and understand complex video content in an event-by-event, easy-to-understand manner. This is based on the observation that natural language expressions often divide a video into distinct, text-related segments, each representing a separate event within a compound event. We introduce EVIS, an Event-Aware Video Instructed Segmentation Assistant, which utilizes text-guided Event Queries to partition a video into simple events, extracting event-aware visual-text features to achieve a hierarchical understanding of the video. Additionally, we propose Object-Pixel-Hybrid Learning, which enables the MLLMs to track targets in long-term videos by integrating fine-grained pixel features with prior object queries. Extensive experimental results on 5 public benchmarks demonstrate EVIS's strong performance in addressing the referring video segmentation task.
