Search papers, labs, and topics across Lattice.
This paper introduces SignSeek, a novel approach for retrieving signs from a dictionary using video queries, addressing the challenge of variability among signers. By employing contrastive learning with saliency-guided articulator masking, SignSeek generates transferable sign representations that excel in open-set retrieval tasks. The model achieves state-of-the-art performance across multiple datasets, including zero-shot generalization to British Sign Language, outperforming existing methods even when they were specifically trained on BSL data.
SignSeek achieves zero-shot generalization to British Sign Language, outperforming models trained specifically on BSL data.
Sign language dictionaries are essential resources for sign language learners, yet automatically retrieving a sign from a dictionary, given only a query video, remains a challenging problem due to the natural variability between signers. Existing sign representation learning methods are built for closed-set recognition, producing embeddings that do not generalise to the open-set, signer-independent setting that retrieval demands. \textbf{SignSeek} closes this gap by contrastively learning sign representations with saliency-guided articulator masking. A contrastive objective aligns same-gloss signs across signers, while our Articulator Saliency-Guided Masking (ASGM) pinpoints the single most critical articulator per sign. This drives two complementary objectives, a masked contrastive alignment (MAC) loss that sees the sign through a single articulator and a masked prediction (MAP) loss that reconstructs it in latent space from the surrounding spatio-temporal context. Pretrained on 266K samples ($\sim$5,700 glosses) across multiple sign languages, \textbf{SignSeek} sets a new state-of-the-art performance in cross-corpus retrieval on ASL-Citizen, WLASL, and NMFs-CSL without any downstream fine-tuning. Strikingly, it achieves zero-shot generalisation to an entirely unseen British Sign Language (BSL), surpassing methods explicitly trained on BSL, and transfers seamlessly to isolated sign recognition and subtitle alignment, outperforming prior skeleton-based methods.
