Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces DiscoSign, a novel framework for translating text to sign language gloss that incorporates discourse-aware processing, addressing critical phenomena such as spatial coreference resolution, Question-Answer Clauses (QACs), and concept-gloss consistency. By leveraging a modular Large Language Model (LLM), DiscoSign significantly enhances spatial consistency and entity tracking in sign language translation compared to traditional sentence-level approaches, while still achieving competitive performance in single-sentence gloss translation. The authors also propose new evaluation metrics tailored to assess discourse coherence, marking a significant advancement in the field of sign language processing.
Discourse-aware translation can drastically improve sign language gloss accuracy, enhancing spatial consistency and entity tracking beyond traditional methods.
Sign language processing systems have traditionally operated at the sentence level, ignoring critical discourse phenomena fundamental to sign language comprehension. We introduce DiscoSign, a computational approach for discourse-aware text to sign language gloss translation grounded in linguistic research. We address three key phenomena within our modular Large Language Model (LLM)-based translation framework: (i) spatial coreference resolution, where entities maintain consistent spatial locations throughout discourse; (ii) Question-Answer Clauses (QACs), pseudocleft structures serving specific discourse functions; and (iii) concept-gloss consistency, ensuring stable mappings between English concepts and American Sign Language (ASL) signs. Traditional translation metrics fail to capture discourse-level quality, so we introduce a suite of novel evaluation metrics designed to assess each dimension of discourse coherence addressed by our framework. Experiments on sentence-level and discourse-level datasets show that our approach for discourse-aware processing significantly improves spatial consistency and entity tracking relative to sentence-only translation, while maintaining competitive single-sentence gloss translation quality. Our work establishes the first systematic framework for discourse-level text to sign language gloss translation with corresponding evaluation methodology.
Apple ML