Search papers, labs, and topics across Lattice.
This paper introduces a gaze-aware multimodal LLM assistant that leverages egocentric video with gaze overlays to infer user cognitive states and provide targeted assistance. A controlled study (n=36) demonstrated that the gaze-aware assistant significantly outperformed a text-only LLM assistant in accuracy, personalization, and user recall. The gaze-aware system also led to more efficient interactions, as evidenced by reduced user speech.
Gaze-tracking unlocks a new level of personalized AI assistance, enabling LLMs to infer user cognitive states and boost recall performance.
Current LLM assistants are powerful at answering questions, but they have limited access to the behavioral context that reveals when and where a user is struggling. We present a gaze-grounded multimodal LLM assistant that uses egocentric video with gaze overlays to identify likely points of difficulty and target follow-up retrospective assistance. We instantiate this vision in a controlled study (n=36) comparing the gaze-aware AI assistant to a text-only LLM assistant. Compared to a conventional LLM assistant, the gaze-aware assistant was rated as significantly more accurate and personalized in its assessments of users'reading behavior and significantly improved people's ability to recall information. Users spoke significantly fewer words with the gaze-aware assistant, indicating more efficient interactions. Qualitative results underscored both perceived benefits in comprehension and challenges when interpretations of gaze behaviors were inaccurate. Our findings suggest that gaze-aware LLM assistants can reason about cognitive needs to improve cognitive outcomes of users.
Microsoft Research
MIT CSAIL