Search papers, labs, and topics across Lattice.
This paper introduces a novel framework for multi-modal global alignment aimed at enhancing self-supervised learning for driver distraction detection, addressing the limitations of conventional contrastive objectives in noisy multi-modal environments. By incorporating soft targets derived from cycle-consistency scores and a similarity-based weighting mechanism, the approach effectively mitigates the impact of unreliable positives and faulty negatives. Evaluated on the Drive&Act dataset, the method demonstrates superior performance over existing baselines, showcasing strong generalization capabilities across diverse camera perspectives.
Faulty negatives and unreliable positives can be effectively managed in multi-modal learning, leading to significant improvements in driver distraction detection accuracy.
Robust self-supervised learning of multi-modal video representations is critical for real-world applications such as driver distraction detection, where multiple sensors provide complementary but noisy signals. Conventional contrastive objectives, such as InfoNCE, assume all negatives are equally informative and all positives are reliable. However, this assumption is frequently violated in multi-modal data due to viewpoint changes, occlusions, or semantic overlap across modalities. In this work, we propose a novel framework for multi-modal global alignment that addresses these challenges by jointly modeling faulty negatives and unreliable or faulty positives. We introduce soft targets derived from cycle-consistency scores to relax the hard-negative assumption, and a weighting mechanism based on similarity distributions to mitigate the impact of noisy or faulty positives. Our approach extends traditional pairwise alignment to a principled global multi-modal setting, aggregating alignment information across all modality pairs. We evaluate our method on the Drive&Act dataset, demonstrating that it consistently outperforms both pairwise and existing global alignment baselines across RGB, IR, Depth, and Skeleton modalities. Cross-view ablation studies further show strong generalization to unseen camera perspectives, highlighting the robustness of our representations. Overall, our framework provides a scalable and effective solution for self-supervised global multi-modal representation learning, enabling reliable driver distraction detection and pioneering in real-world multi-modal video understanding. Our code will be published on GitHub.
