Search papers, labs, and topics across Lattice.
This paper introduces TBSG-Net, a novel Dynamic Scene Graph (DSG) based approach for proposal-free Video Moment Retrieval (VMR) that addresses the limitations of Static Scene Graphs (SSGs) by incorporating temporal dynamics and explicit temporal span encoding. By leveraging a Dynamic Scene Graph Embedding (DSG-E) module, TBSG-Net effectively captures event-centric graph representations, allowing for improved modeling of object interactions over time. Experimental results show that TBSG-Net significantly outperforms existing baselines, highlighting its effectiveness in enhancing spatio-temporal video representations for fine-grained moment retrieval.
TBSG-Net achieves a breakthrough in Video Moment Retrieval by integrating temporal dynamics and explicit duration encoding, leading to superior performance over traditional methods.
Recent advances in proposal-free Video Moment Retrieval (VMR) have highlighted the effectiveness of Static Scene Graphs (SSGs). By modeling objects and their relations at the frame level, SSGs enrich retrieval-oriented video representations. However, integrating SSGs into VMR remains constrained by two inherent limitations: (1) Lack of Temporal Dynamics. SSGs fail to model how objects and their relationships evolve over time, leading to the loss of essential temporal dependencies in video representation; and (2) Lack of Explicit Temporal Span Encoding. SSGs do not explicitly encode the duration of relationships, making precise localization challenging. To address these limitations, we propose Temporal Bipartite Scene Graph Network (TBSG-Net)---to the best of our knowledge, the first Dynamic Scene Graph (DSG) based proposal-free VMR model. Specifically, TBSG-Net leverages DSGs to extract event-centric graph representations of the input video, enabling the modeling of object interactions over time and thus addressing limitation (1). These DSGs are then processed by a novel Dynamic Scene Graph Embedding (DSG-E) module to capture both Temporal Span and spatio-temporal information. First, DSG-E utilizes a TBSG Constructor to transform DSGs into TBSGs, explicitly encoding objects, relationships, and time spans to tackle limitation (2). Second, the resultant TBSGs are passed into a hybrid TBSG Encoder that integrates a Transformer variant for global event modeling and a Graph Convolutional Network for detailed relational reasoning, ultimately producing a more comprehensive spatio-temporal representation. Our experiments demonstrate substantial improvements of TBSG-Net over all baselines.
