Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the zero-shot capabilities of Vision Language Models (VLMs) for Isolated Sign Language Recognition (ISLR) on the WLASL300 benchmark. They find that open-source VLMs significantly underperform traditional supervised ISLR classifiers in a prompt-only zero-shot setting, despite exhibiting some visual-semantic alignment. However, larger proprietary VLMs demonstrate substantially improved accuracy, suggesting that scale and training data play a crucial role in zero-shot ISLR performance.
Turns out, even the best open-source Vision Language Models are still quite deaf when it comes to sign language recognition, but scaling up the model and training data helps it listen.
Recent Vision Language Models (VLMs) have demonstrated strong performance across a wide range of multimodal reasoning tasks. This raises the question of whether such general-purpose models can also address specialized visual recognition problems such as isolated sign language recognition (ISLR) without task-specific training. In this work, we investigate the capability of modern VLMs to perform ISLR in a zero-shot setting. We evaluate several open-source and proprietary VLMs on the WLASL300 benchmark. Our experiments show that, under prompt-only zero-shot inference, current open-source VLMs remain far behind classic supervised ISLR classifiers by a wide margin. However, follow-up experiments reveal that these models capture partial visual-semantic alignment between signs and text descriptions. Larger proprietary models achieve substantially higher accuracy, highlighting the importance of model scale and training data diversity. All our code is publicly available on GitHub.