Search papers, labs, and topics across Lattice.
This study investigates users' mental models of speech translation systems by employing a cross-lingual question answering framework to analyze user interactions with machine translation (MT) outputs. The findings reveal that users improve their predictive accuracy regarding MT errors through practice and familiarity with the source language, largely by utilizing surface-level error cues. Additionally, the inclusion of speech transcriptions enhances users' mental models, highlighting the potential of this approach for advancing human-AI collaboration in translation tasks.
Users can significantly enhance their understanding of machine translation limitations by leveraging surface-level cues and practice, reshaping how we approach user training in AI systems.
Millions of people use machine translation (MT) tools daily, yet little is known about their perception of what systems can and cannot do. This paper studies users' mental models of speech translation systems through a new framework based on cross-lingual question answering, where users either accept MT output or request professional re-translation to answer questions based on the information presented in a foreign language. By analyzing user behavior and accuracy trends across varying translation qualities, we examine to what extent they can predict where the system is likely to be wrong, and how this mental model evolves. Users develop stronger mental models with practice, especially when they have some knowledge of the source language, primarily by relying on surface-level error cues. Moreover, providing speech transcriptions can help users develop better mental models. Our results show the promise of cross-lingual question answering as a downstream task for studying MT mental models and advancing our understanding of human-AI collaboration.
