Search papers, labs, and topics across Lattice.
The authors introduce SUPERGLASSES, a new VQA benchmark designed specifically for AI smart glasses, comprising real-world egocentric data with search trajectories and reasoning annotations. They evaluate 26 VLMs on SUPERGLASSES, revealing substantial performance gaps compared to traditional datasets. To address these gaps, they propose SUPERLENS, a retrieval-augmented agent incorporating object detection, query decoupling, and multimodal web search, achieving state-of-the-art performance and outperforming GPT-4o.
GPT-4o gets outperformed by 2.19% on a new smart glasses VQA benchmark, thanks to a retrieval-augmented agent that first identifies the object of interest.
The rapid advancement of AI-powered smart glasses, one of the hottest wearable devices, has unlocked new frontiers for multimodal interaction, with Visual Question Answering (VQA) over external knowledge sources emerging as a core application. Existing Vision Language Models (VLMs) adapted to smart glasses are typically trained and evaluated on traditional multimodal datasets; however, these datasets lack the variety and realism needed to reflect smart glasses usage scenarios and diverge from their specific challenges, where accurately identifying the object of interest must precede any external knowledge retrieval. To bridge this gap, we introduce SUPERGLASSES, the first comprehensive VQA benchmark built on real-world data entirely collected by smart glasses devices. SUPERGLASSES comprises 2,422 egocentric image-question pairs spanning 14 image domains and 8 query categories, enriched with full search trajectories and reasoning annotations. We evaluate 26 representative VLMs on this benchmark, revealing significant performance gaps. To address the limitations of existing models, we further propose SUPERLENS, a multimodal smart glasses agent that enables retrieval-augmented answer generation by integrating automatic object detection, query decoupling, and multimodal web search. Our agent achieves state-of-the-art performance, surpassing GPT-4o by 2.19 percent, and highlights the need for task-specific solutions in smart glasses VQA scenarios.
