Search papers, labs, and topics across Lattice.
This paper introduces a novel cross-modal feature embedding method that addresses the challenges of face-and-voice association learning by embedding features into a convex hull, thereby enhancing the correlation between voice clips and facial images. The approach incorporates cross-modal attention mechanisms to effectively reduce false positives and negatives, which have plagued previous methods due to inter-modal feature heterogeneity. Experimental results on the VoxCeleb dataset show significant performance improvements in cross-modal verification, matching, and retrieval tasks compared to state-of-the-art techniques.
Embedding face and voice features into a convex hull can drastically reduce false associations, leading to unprecedented accuracy in cross-modal tasks.
Face-and-voice association learning is one of the most challenging tasks in deep learning. In this paper, we propose a simple but powerful cross-modal feature embedding method for the association of faces and voices. Previous work has studied cross-modal association tasks to establish the correlation between voice clips and facial images. These works have addressed cross-modal discrimination but underestimate the importance of handling heterogeneity in inter-modal features between audio and video, resulting in a lot of false positives and false negatives. To tackle the problem, the proposed method learns the embeddings of cross-modal features by making another feature exist between cross-modal features, facilitating the voice and face features of the same person to be embedded in a convex hull. Moreover, the incorporation of cross-modal attention mechanisms with convex embedding techniques represents a highly effective strategy for the attenuation of false positives and false negatives, accomplished via the minimization of inter-class discrepancies. We exhaustively evaluated our method for cross-modal verification, matching, and retrieval tasks on the large-scale VoxCeleb dataset. Extensive experimental results demonstrate that the proposed method achieves notable improvements over existing state-of-the-art methods.
