Search papers, labs, and topics across Lattice.
This paper introduces a reasoning-driven Sign Language Translation (SLT) framework that models translation as a cross-modal reasoning task rather than direct video-to-text conversion. The core of the approach involves an ordered sequence of latent thoughts that progressively extract and organize meaning from the video input. A plan-then-ground decoding strategy is then applied, where the model first plans the translation and subsequently grounds it in the video evidence. Experiments on existing and a newly introduced large-scale gloss-free SLT dataset demonstrate consistent improvements over existing gloss-free methods.
Sign language translation isn't just video-to-text – modeling it as cross-modal reasoning with latent "thoughts" dramatically improves accuracy and coherence.
Many SLT systems quietly assume that brief chunks of signing map directly to spoken-language words. That assumption breaks down because signers often create meaning on the fly using context, space, and movement. We revisit SLT and argue that it is mainly a cross-modal reasoning task, not just a straightforward video-to-text conversion. We thus introduce a reasoning-driven SLT framework that uses an ordered sequence of latent thoughts as an explicit middle layer between the video and the generated text. These latent thoughts gradually extract and organize meaning over time. On top of this, we use a plan-then-ground decoding method: the model first decides what it wants to say, and then looks back at the video to find the evidence. This separation improves coherence and faithfulness. We also built and released a new large-scale gloss-free SLT dataset with stronger context dependencies and more realistic meanings. Experiments across several benchmarks show consistent gains over existing gloss-free methods. Code and data will be released upon acceptance at https://github.com/fletcherjiang/SignThought.
