Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To resolve performance degradation from domain shifts and missing sensor data in active speaker detection, ROAM-ASD jointly models audio, full-face, and fine-grained mouth features within a unified multi-stream architecture. By employing joint self-attention with modality-agnostic query tokens and structured modality dropout, the network dynamically adapts to missing or corrupted observations without specialized fallback sub-networks. The method sets new state-of-the-art benchmarks across five evaluation suites—including 98.8% mAP on WASD (+5.1) and 87.9% on UniTalk (+4.7)—while exhibiting robust zero-shot cross-dataset generalization.
Missing sensory streams typically break audiovisual tracking, but routing audio, face, and mouth inputs through modality-agnostic query attention shatters previous active speaker detection records across five benchmarks by up to 5.1 mAP.
Active speaker detection (ASD) requires reliable association between visible faces and acoustic speech, yet existing systems often degrade under challenging domains or incomplete observations. We introduce ROAM-ASD, a robust audiovisual framework that jointly models audio, full-face, and fine-grained mouth representations. A unified joint self-attention mechanism processes all input streams together with modality-agnostic query tokens, enabling direct interaction among available modality inputs. Modality dropout further improves robustness when input streams are unavailable. ROAM-ASD achieves state-of-the-art performance across five ASD benchmarks: 98.8% mAP on WASD, 87.9% on UniTalk, 96.5% on AVA, 99.3% on ASW, and 98.2% on Talkies, improving over previous best systems by 5.1, 4.7, 0.9, 1.0, and 2.1 mAP points, respectively. ROAM-ASD also substantially improves zero-shot cross-dataset generalization and remains robust to missing observations.