Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To enable real-time communication across disparate sign languages, this work establishes the first simultaneous sign-to-sign translation framework utilizing both test-time wait-k inference and streaming models trained via stochastic multi-path supervision. Existing sign-to-sign architectures rely on offline, full-sequence inputs, failing under live constraints such as video conferencing where source-target syntactic divergence severely complicates streaming generation. Evaluated across six translation directions, the streaming policy cuts computation-aware latency (ca-Stream-AL) by 38% while incurring only a 2.1 BLEU-4 drop and keeping kinematic joint error (DTW-PA-MPJPE) within 9% of full-sentence offline baselines.
Real-time translation between different sign languages is finally feasible, cutting streaming latency by 38% with only a 2.1 BLEU penalty against offline models despite severe cross-lingual word-order mismatches.
Deaf and hard-of-hearing (DHH) signers cannot converse in real time across different sign languages today: existing sign-to-sign translation systems run offline, requiring the full source clip before any target sign is emitted. Live use cases - e.g. broadcast interpretation and two-way video calls - instead demand simultaneous output, while the source signer is still signing. We present, to our knowledge, the first simultaneous sign-to-sign (S2S) translation system, with two wait-k regimes: test-time wait-k inference applied directly to a full-sentence model, and a trained wait-k model via stochastic multi-path supervision. We further introduce ca-Stream-AL, a computation-aware latency metric for streaming output. Averaged across six S2S directions on both a smaller human-verified test set and a larger synthetic S2S corpus, our streaming system achieves a 38% ca-Stream-AL reduction while staying within a 9% DTW-PA-MPJPE increase and a 2.1 BLEU-4 drop compared to the full-sentence baseline. A word-order case study probes how the streaming model handles word order mismatch between different sign languages - a consequence of simultaneous translation.