Search papers, labs, and topics across Lattice.
The paper introduces Egocentric Co-Pilot, a web-native neuro-symbolic framework for smart glasses that leverages LLMs to orchestrate perception, reasoning, and web tools for assistive egocentric AI. It combines Temporal Chain-of-Thought with Hierarchical Context Compression for long-horizon question answering from first-person video, and uses a multimodal intent layer to map speech and gaze to structured commands. Experiments on Egolife and HD-EPIC datasets, along with a user study, demonstrate competitive egocentric QA performance and improved task completion compared to commercial baselines.
Smart glasses powered by web-native AI agents can now outperform commercial solutions in assistive tasks, offering a practical path to always-on, context-aware help for users navigating daily life.
What if accessing the web did not require a screen, a stable desk, or even free hands? For people navigating crowded cities, living with low vision, or experiencing cognitive overload, smart glasses coupled with AI agents could turn the web into an always-on assistive layer over daily life. We present Egocentric Co-Pilot, a web-native neuro-symbolic framework that runs on smart glasses and uses a Large Language Model (LLM) to orchestrate a toolbox of perception, reasoning, and web tools. An egocentric reasoning core combines Temporal Chain-of-Thought with Hierarchical Context Compression to support long-horizon question answering and decision support over continuous first-person video, far beyond a single model's context window. Additionally, a lightweight multimodal intent layer maps noisy speech and gaze into structured commands. We further implement and evaluate a cloud-native WebRTC pipeline integrating streaming speech, video, and control messages into a unified channel for smart glasses and browsers. In parallel, we deploy an on-premise WebSocket baseline, exposing concrete trade-offs between local inference and cloud offloading in terms of latency, mobility, and resource use. Experiments on Egolife and HD-EPIC demonstrate competitive or state-of-the-art egocentric QA performance, and a human-in-the-loop study on smart glasses shows higher task completion and user satisfaction than leading commercial baselines. Taken together, these results indicate that web-connected egocentric co-pilots can be a practical path toward more accessible, context-aware assistance in everyday life. By grounding operation in web-native communication primitives and modular, auditable tool use, Egocentric Co-Pilot offers a concrete blueprint for assistive, always-on web agents that support education, accessibility, and social inclusion for people who may benefit most from contextual, egocentric AI.
