Search papers, labs, and topics across Lattice.
This paper introduces Sign Language Question Answering (SLQA), a novel task designed to evaluate sign language understanding by requiring models to answer arbitrary natural language questions about sign language videos. To support this task, the authors create two benchmarks, SignQA, which include a diverse set of question categories and are generated from existing annotations, thus providing a more comprehensive assessment of model capabilities. The proposed baseline model, featuring a Question-Conditioned Modulated Temporal Downsampling module, outperforms existing vision-language models, setting a new standard for evaluating sign language understanding.
SLQA reveals that models can achieve deeper comprehension of sign language by answering complex questions, rather than merely translating or recognizing signs.
Recent advances in sign language (SL) understanding (SLU) have led to remarkable progress in tasks such as continuous SL recognition and SL translation. However, these tasks are designed with predefined objectives, requiring models to learn a fixed mapping from sign videos to glosses or spoken-language sentences. As a result, they provide only a limited assessment of whether a model truly understands the semantic content of SL videos. To address this limitation, \textbf{we first propose a new task, Sign Language Question Answering (SLQA)}, which evaluates SL understanding by requiring models to answer arbitrary natural language questions about SL videos. Unlike previous SLU tasks, SLQA provides a more flexible and comprehensive evaluation framework that assesses multiple reasoning capabilities beyond recognition and translation. To facilitate this task, \textbf{we further construct two SignQA benchmarks} based on PHOENIX14T and CSL-Daily by automatically generating question-answer pairs from existing gloss and sentence annotations using carefully designed templates. The resulting datasets cover five complementary question categories, including position reasoning, structural reasoning, visual search, gloss recognition, and translation understanding. \textbf{Finally, we propose a simple yet effective baseline model} equipped with a Question-Conditioned Modulated Temporal Downsampling module and an in-domain knowledge transfer strategy, enabling effective knowledge transfer from existing SLU tasks while enhancing question-aware temporal feature modeling. Extensive experiments demonstrate that our baseline consistently outperforms representative vision-language models across all question categories, establishing a strong benchmark for future research on SLQA. Datasets are available at:{https://huggingface.co/datasets/hulala/SignQA-2026}.
