Search papers, labs, and topics across Lattice.
This study explores the adaptation of Voice Activity Projection (VAP) for predictive turn-taking in sign language interactions, addressing a significant gap in social robot communication capabilities. By utilizing interaction recordings from the Public DGS Corpus, the researchers derive binary signing activity streams and employ pose-derived features to predict turn-taking dynamics. The findings indicate that while SHIFT/HOLD predictions show promise, SHIFT-prediction remains challenging, highlighting both the potential and limitations of transferring spoken interaction models to sign language contexts.
Predictive turn-taking in sign language interactions reveals that hand cues significantly enhance SHIFT/HOLD predictions, but SHIFT remains elusive.
Social robots must interact robustly not only with users assumed by speech-centered systems but also with diverse users whose communication relies on different modalities, e.g., sign language. One important capability gap is predictive turn-taking with signing users. Although Voice Activity Projection (VAP) has been successfully used to model future voice activity in spoken interaction, it remains unclear whether the framework transfers to sign language interaction. This paper presents an initial transfer study of adapting a VAP architecture to dyadic sign language interaction. Using interaction recordings from the Public DGS Corpus, we derive binary signing activity streams from lexical sign annotations and formulate proxy tasks for turn-taking prediction. The model uses pose-derived hand, eye-region, and mouth-region features extracted for each signer. The results show that SHIFT/HOLD prediction is promising, especially with hand cues, while SHIFT-prediction remains difficult. These findings provide initial evidence for both the promise and the current limitations of transferring predictive turn-taking models from spoken interaction to sign language interaction. Predictive modeling of sign language interaction still requires sign-language-specific event definitions that go beyond speech-derived categories.
