Search papers, labs, and topics across Lattice.
SignDPO introduces a multi-level Direct Preference Optimisation framework aimed at improving skeleton-based Sign Language Translation by enhancing alignment across spatial, temporal, and linguistic dimensions. By shifting the optimization focus from mere sequence mimicry to structured preference alignment, the framework addresses the limitations of existing imitation-based models that often suffer from semantic drift. Extensive evaluations on benchmarks like CSL-Daily, How2Sign, and OpenASL reveal that SignDPO not only surpasses state-of-the-art gloss-free methods but also competes effectively with gloss-based approaches.
Multi-level preference alignment in SignDPO significantly reduces semantic drift, outperforming traditional gloss-free models and challenging gloss-based benchmarks.
We present SignDPO, a novel multi-level Direct Preference Optimisation (DPO) framework designed to enhance the alignment of skeleton-based Sign Language Translation. While current skeleton-based models have made significant progress using Maximum Likelihood Estimation, they are primarily constrained by an imitation-based paradigm that lacks discriminative sensitivity to the fine-grained spatio-temporal nuances of sign language, often leading to semantic drift. To address this, SignDPO shifts the optimisation goal from simple sequence mimicry to structured preference alignment across spatial, temporal, and linguistic dimensions. Our framework involves three key designs. First, we introduce a hierarchical perturbation strategy to construct spatial and temporal non-preferred samples at both global and local granularities automatically. Second, we propose a self-guiding mechanism that leverages decoder cross-attention scores to identify and perturb semantically salient skeletal regions, forcing the model to distinguish genuine sign signals from structural distortions. Third, we establish an automated language-level preference generator by fine-tuning a dedicated perturbation model, capturing complex output-level failure modes without manual annotation. Extensive experiments on three widely adopted benchmarks, CSL-Daily, How2Sign, and OpenASL, demonstrate that SignDPO consistently outperforms state-of-the-art gloss-free methods and even rivals established gloss-based ones. Our results suggest that multi-level preference alignment is a powerful paradigm for bridging the gap between high-entropy skeletal trajectories and discrete linguistic semantics.
