Search papers, labs, and topics across Lattice.
This paper introduces the Missing-Token Prompted Reliability-Aware Fusion (MRAF) framework, which enhances polyglot speaker identification by addressing the challenges of language generalization and reliability in the absence of face information. MRAF employs a learnable missing token to represent unavailable face inputs, facilitating a unified token space for reliability estimation and cross-modal fusion. Experimental results on the POLY-SIM 2026 test set show that MRAF achieves 100% accuracy in standard conditions and competitive performance in more challenging scenarios, highlighting its robustness and effectiveness.
A learnable missing token representation allows MRAF to achieve 100% accuracy in polyglot speaker identification, even when face information is missing.
Accurate and robust multimodal speaker identification is essential for multimedia understanding and biometric authentication. However, real-world polyglot scenarios pose two key challenges: speaker-discriminative representations should generalize across languages, and the model should remain reliable when face information is unavailable. To address these challenges, we propose MRAF, a Missing-Token Prompted Reliability-Aware Fusion framework for polyglot speaker identification across complete-modality, missing-face, and cross-lingual scenarios. MRAF represents unavailable face inputs with a learnable missing token instead of fixed zero-valued features, providing a trainable representation of the missing visual state. This design reduces the distribution gap caused by missing inputs and allows subsequent reliability estimation and cross-modal fusion to operate within a unified token space. To adaptively integrate modalities with different reliability, MRAF further introduces a reliability-aware cross-attention fusion module, which estimates face and audio reliability scores, normalizes them into modality weights, and applies these weights to token representations before bidirectional cross-attention. In this way, the model can emphasize reliable modality cues while suppressing unreliable ones. During training, MRAF jointly optimizes multi-branch classification losses, audio-only knowledge distillation, and center loss to improve speaker discrimination and missing-modality robustness. Experiments on the official POLY-SIM 2026 test set demonstrate the effectiveness of the proposed framework. In the final evaluation, MRAF achieves 100% accuracy on P3 and P5, and obtains competitive results on the more challenging missing-face settings P4 and P6. The source code will be released at https://github.com/MSA-LMC/MRAF.
