Search papers, labs, and topics across Lattice.
The paper introduces IPIBench, a new benchmark designed to evaluate the interactive proactive intelligence of multimodal large language models (MLLMs) in dynamic, multi-turn streaming video scenarios. The benchmark assesses proactive monitoring, task management, and interleaved reactive-proactive requests, revealing limitations in existing MLLMs regarding stable proactive triggering and reactive-proactive coordination. To address these limitations, the authors propose IPI-Agent, a training-free agentic framework that incorporates an interaction-control policy and temporal-gating mechanism, demonstrating consistent performance improvements across the benchmark.
MLLMs struggle to juggle proactive tasks and reactive queries in dynamic video streams, but a simple agentic framework can significantly improve their coordination without any training.
Recent multimodal large language models (MLLMs) achieve strong performance on reactive question answering, but real-world streaming assistants require proactive reasoning over continuous visual inputs. Existing benchmarks mainly study reactive or proactive interactions in isolated single-turn settings, overlooking dynamic multi-turn scenarios where users may add, modify, or cancel proactive requests alongside interleaved reactive queries. To address this gap, we introduce IPIBench, the first benchmark for evaluating Interactive Proactive Intelligence of MLLMs under streaming video settings. IPIBench covers proactive monitoring, proactive task management, and interleaved reactive-proactive requests. Evaluations on representative MLLMs reveal two major limitations: unstable proactive triggering and weak coordination between reactive and proactive behaviors. We further propose IPI-Agent, a training-free agentic framework with an interaction-control policy and a temporal-gating mechanism for stabilizing proactive triggering and coordinating multi-turn interactions. Experiments show that IPI-Agent consistently improves existing MLLMs across all benchmark settings.
Tsinghua AI