Search papers, labs, and topics across Lattice.
This paper introduces OmniAssistBench, a benchmark designed to evaluate omni-modal large language models (Omni-LLMs) in interactive assistant roles, where they must combine visual inputs, user goals, and prior knowledge. By reverse-engineering existing Internet videos to create a dataset that simulates continuous interactions, the authors address the challenges posed by dynamic user interactions that traditional static datasets cannot capture. Results reveal that while models like Gemini-3-Pro and Qwen3-Omni-Instruct show promise, they still struggle with visual prompts and maintaining context, highlighting significant areas for improvement in their performance as reliable assistants.
Current omni-modal models can score over 66 points in interactive tasks, but they still falter on visual cues and context retention, revealing a critical gap in their utility as assistants.
Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants, which continuously perceive environments and guide users to achieve specific goals. Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help. Evaluating this is rather challenging, as the model's unpredictable response dynamically changes the user's subsequent actions, which static offline datasets cannot accommodate. To address this bottleneck, we introduce OmniAssistBench. To solve the issue of diverging interaction paths where the same user goal can be achieved through various methods, we provide models with predefined priors derived from the source video, requiring them to guide users along the exact same routes. Since real interaction videos are rare, we construct the dataset by reverse-engineering existing Internet videos. We deduce logical user goals and segment the videos into multi-turn clips to simulate continuous interactions. This rigorous pipeline required over 1000 expert person-hours to build the dataset. Results show that the proprietary Gemini-3-Pro reaches 66.4 out of the max point of 100, while the open-source Qwen3-Omni-Instruct achieves 51.2. Although current models generally understand user inputs, they frequently provide incorrect or incomplete answers. Specifically, they struggle with visual prompts (e.g., hand gestures), fail to maintain historical context during multi-turn interactions, and fail to delay response until the target event. Results indicate substantial room for improvement before models can become reliable assistants.
