Search papers, labs, and topics across Lattice.
This paper introduces Motion-guided Semantic Alignment (MoSA), a new method for Dynamic Scene Graph Generation (DSGG) that enhances fine-grained relationship modeling by incorporating motion attributes and aligning visual features with semantic embeddings. MoSA uses a Motion Feature Extractor (MFE) to encode object-pair motion attributes, which are then fused with spatial relationship features via a Motion-guided Interaction Module (MIM). The method also employs a cross-modal Action Semantic Matching (ASM) mechanism and a category-weighted loss to improve semantic discrimination and tail relationship learning, achieving state-of-the-art results on the Action Genome dataset.
Motion cues and semantic alignment can significantly boost dynamic scene graph generation, especially for modeling nuanced relationships between objects in videos.
Dynamic Scene Graph Generation (DSGG) aims to structurally model objects and their dynamic interactions in video sequences for high-level semantic understanding. However, existing methods struggle with fine-grained relationship modeling, semantic representation utilization, and the ability to model tail relationships. To address these issues, this paper proposes a motion-guided semantic alignment method for DSGG (MoSA). First, a Motion Feature Extractor (MFE) encodes object-pair motion attributes such as distance, velocity, motion persistence, and directional consistency. Then, these motion attributes are fused with spatial relationship features through the Motion-guided Interaction Module (MIM) to generate motion-aware relationship representations. To further enhance semantic discrimination capabilities, the cross-modal Action Semantic Matching (ASM) mechanism aligns visual relationship features with text embeddings of relationship categories. Finally, a category-weighted loss strategy is introduced to emphasize learning of tail relationships. Extensive and rigorous testing shows that MoSA performs optimally on the Action Genome dataset.
