Search papers, labs, and topics across Lattice.
This paper introduces Compositional Spatio-Temporal Video Grounding (CompSTVG), addressing the inadequacies of current models in handling complex, compositional queries that are essential for real-world video grounding tasks. By developing a synthetic data engine that generates difficulty-graded training and evaluation datasets, the authors create STVG-CompBench, a benchmark that highlights the significant performance drop of existing models on these challenging queries. The proposed curriculum reinforcement learning framework, CurrSTVG, demonstrates substantial improvements in model performance, particularly on the most difficult compositional tasks.
Current spatio-temporal video grounding models falter on complex queries, with performance dropping sharply when faced with compositional challenges.
Despite the impressive progress of recent MLLMs on spatio-temporal video grounding (STVG), existing evaluations and training data focus primarily on simple queries. They largely overlook the compositional queries prevalent in real-world scenarios, where a target must be disambiguated by jointly reasoning about its attributes and relations to other entities. To bridge this gap, we propose Compositional Spatio-Temporal Video Grounding (CompSTVG), a task that requires models to process complex textual queries where every intertwined attribute and relational cue is essential for disambiguation. To facilitate this task at scale, we build a synthetic data engine that leverages a spatio-temporal scene graph as a difficulty measure and casts difficulty-controlled query synthesis as a constraint programming problem, producing difficulty-graded data for both evaluation and training. Built on this engine, we introduce STVG-CompBench, a benchmark stratified by explicit difficulty levels that jointly capture temporal complexity and spatial interference. Evaluating 11 representative STVG models on STVG-CompBench reveals that current models perform poorly on compositional queries, exhibiting a sharp performance drop that is typically obscured by overall dataset-level averages. We further construct synthetic training data and propose CurrSTVG, a curriculum reinforcement learning framework that delivers consistent gains, with the largest improvements observed on the most challenging compositional queries.
