Search papers, labs, and topics across Lattice.
This paper introduces Q-BridgeNet, a novel framework for multilingual sign language translation (SLT) that addresses the challenges of cross-lingual conflicts by utilizing a unified model for both sign and spoken languages. The approach employs discrete Q-units through adaptive segmentation and residual vector quantization, allowing for the capture of shared semantics while accommodating language-specific variations. Experimental results on multiple datasets demonstrate that Q-BridgeNet achieves state-of-the-art performance on native sign-spoken pairs and shows significant generalization capabilities for non-native pairs.
Q-BridgeNet not only sets a new benchmark for native sign-spoken translation but also excels in translating across diverse sign languages with minimal cross-lingual conflicts.
Most sign language translation (SLT) methods focus on isolated native sign-spoken pairs (e.g., American Sign Language - English). Extending language-specific SLT models to multilingual translation would improve accessibility by enabling communication across diverse sign and spoken language communities. However, existing multilingual SLT approaches still struggle to learn a unified model that minimizes cross-lingual conflicts while capturing shared cross-lingual semantics and preserving language-specific variations across different sign languages. Therefore, we propose Q-BridgeNet, a unified framework for multilingual SLT that jointly mitigates cross-lingual conflicts across both the sign language and spoken language sides. On the sign language side, Q-BridgeNet learns discrete Q-units via adaptive segmentation and residual vector quantization: a shared base codebook provides language-agnostic semantic primitives, while language-specific residual codebooks refine heterogeneous signing semantics. On the spoken language side, a multilingual LLM is fine-tuned to operate in the Q-unit space, leveraging cross-lingual priors to enable a unified SLT model. Experiments on PHOENIX14T, How2Sign, and CSL-Daily show that Q-BridgeNet effectively mitigates cross-lingual conflicts, achieving state-of-the-art performance on native sign-spoken pairs while also demonstrating strong generalization to non-native pairs. Our source code is publicly available at: https://github.com/FengLiQ/Q-BridgeNet
Meta AI