Search papers, labs, and topics across Lattice.
This paper introduces a visually grounded method for mapping spoken words to their written counterparts using only images and their spoken descriptions, circumventing the need for explicit textual supervision. By leveraging image captioning systems to create a vocabulary of written words and employing unsupervised word discovery techniques, the authors successfully align spoken utterances with their corresponding written forms. The approach not only surpasses a strong neural baseline in spoken word retrieval and keyword spotting tasks but also offers greater interpretability, highlighting its potential for application in low-resource languages lacking transcripts.
Achieving spoken word segmentation without any text supervision, this method outperforms traditional neural approaches while enhancing interpretability.
How can we learn the mapping between written words and their spoken counterparts in the absence of explicit textual supervision? We present a visually grounded method for building a vocabulary of spoken words using only images and their spoken descriptions. First, image captioning systems are used to build a vocabulary of written words representing salient visual concepts in the images. For each word, we then find utterances whose image captions contain that word. Then we use an unsupervised word discovery technique to align these utterances to locate instances of the target word. The result is spoken word segments that are linked to written words -- all accomplished without any text supervision. In spoken word retrieval and keyword spotting experiments, the proposed approach outperforms a strong neural baseline while being more interpretable. These results demonstrate the feasibility of the approach in English and motivate future work on low-resource languages without transcripts.
