Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces a novel corpus augmentation technique for sign language translation (SLT) that leverages existing gloss-annotated training data and a large language model (LLM) to generate synthetic video-text pairs without requiring additional human annotation or external resources. By extracting per-gloss clips from training videos and generating novel gloss-sentence pairs, the authors significantly enhance the training dataset, leading to a notable improvement of +2.92 BLEU-4 over the baseline without altering the underlying architecture or training protocols. The findings also reveal that while synthetic data can enhance SLT objectives, it may negatively impact vision-language pretraining, suggesting that abrupt transitions in synthetic sequences could serve as a form of implicit regularization.
Achieving a +2.92 BLEU-4 improvement in sign language translation accuracy without any changes to model architecture or training methods highlights the power of corpus augmentation using LLMs.
Sign language translation (SLT) converts sign language video into spoken language text and holds significant promise for improving accessibility and enabling communication between signing and non-signing communities. While large weakly-aligned datasets have enabled pre-training at scale and gloss-free methods have reduced reliance on expert annotation, high-quality parallel sign video-text pairs for fine-tuning remain scarce, limiting generalisation on long-tail vocabulary and unseen constructions. We propose a corpus augmentation approach that requires no additional human annotation, external sign-language video corpora, or generative video models, relying only on the existing gloss-annotated training corpus and an LLM for sentence generation: per-gloss clips are extracted from training videos via CTC forced-alignment, novel gloss-sentence pairs are generated by a corpus-anchored LLM, and synthetic sequences are assembled through random sentence sampling and clip assignment. The resulting synthetic RGB video-text pairs are architecture-agnostic at the downstream training stage and can be consumed directly by RGB-based SLT models, or converted into pose or feature representations by pipelines that derive such inputs from video. Sincan et al. re-evaluated five recent gloss-free methods under strictly identical conditions; the largest verified gain over the GFSLT-VLP baseline was only 0.98 BLEU-4. Our augmentation, applied within the same framework, achieves +2.92 BLEU-4 without any change to architecture or training protocol. We further identify that synthetic data harms vision-language pretraining despite improving its objectives, and that optimising clip transitions for visual smoothness is counter-productive under L2-based criteria; we propose that abrupt boundaries may act as a form of implicit regularisation. Code is available at https://github.com/robizso/slt-datagen.