Search papers, labs, and topics across Lattice.
This paper introduces FATE, a novel audio-visual model that effectively captures both semantic meaning and temporal alignment by retaining frame-level sequences and computing similarity over aligned frame pairs. By combining cross-video semantic learning with within-video temporal contrastive learning, FATE addresses the limitations of existing models that either lose temporal information or fail to understand semantics. The results demonstrate that FATE significantly outperforms strong baselines in temporal and semantic retrieval tasks and achieves competitive performance in event localization, aligning closely with human judgment metrics.
FATE not only retains temporal information but also encodes synchronization in a reusable embedding space, outperforming existing models in both semantic and temporal tasks.
When a dog opens its mouth and barks, humans naturally recognize what the sound is and when it occurs. Building audio-visual models with this same ability requires representations that capture both semantic and temporal alignment. Current approaches fall short on one side or the other: embedding models match semantic but lose temporal information; synchronization models capture temporal offsets but lack semantic understanding. To bridge this gap, we propose FATE, Frame-level Audio-visual Temporal Embedding. Unlike prior embedding models that pool each modality into a single embedding and discard temporal information, FATE retains frame-level sequences, aligns them on the physical timeline, and computes similarity over strictly aligned frame pairs. Unlike synchronization models that output only an offset prediction, FATE encodes synchronization in a reusable embedding space, trained with a joint objective combining cross-video semantic and within-video temporal contrastive learning to capture both what sounds and when it occurs. Across three tasks, FATE surpasses the strongest baseline on temporal and semantic retrieval by a large margin, matches fully supervised methods on event localization in a zero-shot setting, and achieves the best correlation with human judgments as a generation evaluation metric. The source code can be found at https://github.com/guankaisi/FATE.
