Search papers, labs, and topics across Lattice.
This study explores an alignment-based approach to map written English keywords to their spoken Hindi counterparts using visually grounded speech data. By leveraging self-supervised speech representations and off-the-shelf image captioning systems, the authors align Hindi utterances with English tags, aggregating evidence to identify consistent speech segments. The results indicate that this method outperforms previous end-to-end multimodal neural models, highlighting the effectiveness of using negative examples in the alignment process.
Cross-lingual word-to-speech mappings can be effectively learned from visual grounding without the need for transcriptions or extensive model training.
In many low-resource settings, even just eliciting speech for data collection is difficult. One promising approach has been to ask speakers to describe images. But how do we build models from such visually grounded speech data? Given a dataset of images with Hindi spoken captions, we consider how we can map a written English keyword to spoken realisations of that word in Hindi. Previous work trained end-to-end multimodal neural models. Instead, we explore a simpler alignment-based approach built on self-supervised speech representations. Written English tags are automatically obtained from images using off-the-shelf image captioning systems. Hindi utterances associated with the same keyword are then aligned (using self-supervised features), and alignment evidence is aggregated to identify recurring speech segments corresponding to the target word. Experiments evaluating keyword spotting and localization show that our alignment-based approach outperforms a previous attention-based neural model. We also show the benefit of incorporating negative examples during alignment. Our work demonstrates that cross-lingual word-to-speech mappings can be learned directly from visual grounding without transcriptions or explicit model training.
