Search papers, labs, and topics across Lattice.
This study introduces SafeGesture, a benchmark designed to assess the ability of vision-language models (VLMs) to interpret hand gestures in safety-critical contexts by pairing gestures with operational scenarios. The evaluation of models like GPT-4o and Qwen2.5-VL reveals a significant decoupling between gesture recognition accuracy and the ability to derive appropriate safety actions, with GPT-4o achieving high gesture accuracy (98.4%) but only 53.3% in safety accuracy. The findings highlight that the primary challenge lies in scenario-conditioned safety reasoning rather than gesture recognition, as evidenced by the performance gaps and the surprising effectiveness of a scenario-majority policy without visual input.
Despite high gesture recognition rates, leading VLMs struggle with safety reasoning, exposing critical gaps in their operational reliability.
Open-weight and frontier vision-language models (VLMs) perform well on general image understanding, but their ability to interpret fine-grained hand gestures in safety-critical operational contexts remains largely unexamined. We introduce SafeGesture, a benchmark that evaluates whether a model can infer scenario-appropriate safety actions from hand gestures. It pairs six HaGRID gestures with eight operational scenarios for 4,800 items and evaluates Qwen2.5-VL-7B, LLaVA-NeXT-7B, InternVL2-8B, Phi-3.5-Vision, and GPT-4o. Results reveal a perception-reasoning decoupling: GPT-4o achieves 98.4% gesture accuracy but 53.3% safety accuracy, while Qwen2.5-VL reaches 84.9% and 39.5%, yielding gaps of 45.0 and 45.4 percentage points. Four of five models rarely or never use the uncertainty label, and failure directions differ substantially across models. Accuracy also obscures label bias: a scenario-majority policy with no visual input reaches 58.3%, above every evaluated model, while only GPT-4o exceeds this prior under macro-F1. Visual input improves safety accuracy by 11.2 to 30.2 percentage points, but providing the ground-truth gesture as text improves performance by only 0.4 to 3.2 points, and no model exceeds 56.2%. These results indicate that the main bottleneck is scenario-conditioned safety reasoning rather than gesture recognition.
