Search papers, labs, and topics across Lattice.
This paper introduces DualAnchor, a novel gloss-free training framework for sign language translation (SLT) that addresses language-prior degradation and lexical fidelity gaps in existing LLM-based methods. By integrating Token-level Prior Anchoring (TPA) to preserve language priors and Optimal Transport Alignment (OTA) for enhanced lexical fidelity, the framework significantly improves the fluency and accuracy of translations. Experimental results on the PHOENIX-2014T and CSL-Daily datasets demonstrate that DualAnchor outperforms previous approaches, with targeted analyses revealing that TPA enhances fluency while OTA minimizes lexical errors.
DualAnchor not only preserves language priors but also bridges the lexical fidelity gap in sign language translation, leading to significantly improved translation quality.
Recent advances in large language models (LLMs) have led sign language translation (SLT), the task of converting sign-language videos into spoken-language text, to increasingly adopt LLMs as textual backbones. However, despite their strong language modeling capabilities, existing LLM-based SLT methods often undermine rather than exploit this language prior, producing disfluent translations, a failure we term language-prior degradation. Meanwhile, existing methods typically align videos and text at the sentence level, which does not ensure accurate lexical details and creates a lexical fidelity gap. To address both issues, we propose DualAnchor, a gloss-free LLM-based SLT training framework that couples two complementary anchors for linguistically fluent and visually faithful generation. Token-level Prior Anchoring (TPA) preserves the LLM's language prior by regularizing the multimodal decoder at each decoding step toward the next-token distribution of a frozen LLM conditioned on the same autoregressive prefix. Optimal Transport Alignment (OTA) improves lexical fidelity by formulating visual-textual matching as entropy-regularized partial optimal transport, with Sinkhorn optimization inducing a soft alignment between visual tokens and textual content tokens under a cosine cost. DualAnchor achieves strong overall performance on both PHOENIX-2014T and CSL-Daily. Targeted analyses attribute these gains to the complementary effects of the two anchors: TPA improves fluency, whereas OTA reduces fine-grained lexical errors.
