Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
SignFLIP unifies sign language translation (SLT) and generation (SLG) into a single LLM-centered, symmetric architecture trained via stage-wise alignment on large-scale data. By progressively refining shared representations—leveraging initial cross-modal pre-alignment for SLT, which in turn reinforces SLG—the model circumvents the fragmentation of prior task-specific systems. Across multiple standard benchmarks, the framework achieves parity with or outperforms specialized baselines on both translation and generation, while transferring robustly to sign language recognition.
Unifying sign translation and generation within a single LLM reveals that representations adapted for translation directly boost generative fidelity, matching bespoke task-specific architectures across the board.
Sign language translation and generation share the goal of bidirectional alignment between text and sign representations. However, existing approaches either treat them as isolated tasks or are only verified on limited datasets, limiting effective modeling between modalities. In this paper, we propose SignFLIP, a unified LLM-centered framework for translation and generation. To enable bidirectional mapping between text and sign, SignFLIP adopts a symmetric architecture together with a stage-wise training strategy built on large-scale data. The shared sign--text representation is progressively refined: pre-alignment facilitates subsequent SLT, while the SLT-adapted representation further benefits SLG. Extensive experiments on multiple benchmarks show that SignFLIP shows competitive performance compared with task-specific models on both translation and generation tasks, as well as strong transferability to sign language recognition.