Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To test whether modern sign language translation (SLT) models genuinely generalize to new users, this work audits three leading gloss-free models (GFSLT-VLP, GASLT, and SignCL) on PHOENIX14T and CSL-Daily using signer-fold cross-validation. Under strict signer-independent evaluation, translation performance collapses dramatically, with GFSLT-VLP plummeting from 21.44 to 3.59 BLEU-4 and SignCL dropping from 22.74 to 3.66 BLEU-4 on PHOENIX14T. The findings demonstrate that standard SLT benchmarks drastically overestimate capability by rewarding signer-specific memorization and train-test target sentence leakage rather than true language translation.
Leading sign language models lose over 80% of their BLEU score the moment they encounter an unseen signer, exposing widespread benchmark inflation driven by identity and sentence leakage.
Sign Language Translation has advanced with deep learning, yet evaluations remain largely signer-dependent, with overlapping signers across train/dev/test. This raises concerns about whether models truly generalise or instead rely on signer-specific regularities. We conduct signer-fold cross-validation on GFSLT-VLP, GASLT, and SignCL, three leading, publicly available, gloss-free SLT models, on CSL-Daily and PHOENIX14T. Under signer-independent evaluation, performance drops sharply: on PHOENIX14T, GFSLT-VLP falls from BLEU-4 21.44 to 3.59 and ROUGE-L 42.49 to 11.89; GASLT from 15.74 to 8.26; and SignCL from 22.74 to 3.66. We also observe that in CSL-Daily many target sentences are performed by multiple signers, so common splits can place identical sentences in both training and test, inflating absolute scores by rewarding recall of recurring sentences rather than genuine generalisation. These findings indicate that signer-dependent evaluation can substantially overestimate SLT capability. We recommend: (1) adopting signer-independent protocols to ensure generalisation to unseen signers; (2) restructuring datasets to include explicit signer-independent, sentence-disjoint splits for consistent benchmarking; and (3) reporting both signer-dependent and signer-independent results together with train-test sentence overlap to improve transparency and comparability.