Search papers, labs, and topics across Lattice.
The paper introduces PIRA-Bench, a new benchmark for evaluating multimodal large language models (MLLMs) on proactive intent recommendation from continuous GUI screenshots. This benchmark addresses the limitations of existing reactive GUI agent datasets by featuring complex, long-horizon trajectories with interleaved intents, noisy segments, and varying user profiles. To facilitate research on this benchmark, the authors also propose PIRF, a memory-aware, state-tracking framework that enables MLLMs to handle multiple task threads and misleading visual inputs.
Current GUI agents are reactive, but PIRA-Bench offers a challenging new environment for training agents to *proactively* anticipate user intentions from continuous visual inputs, a crucial step towards truly intelligent AI assistants.
Current Graphical User Interface (GUI) agents operate primarily under a reactive paradigm: a user must provide an explicit instruction for the agent to execute a task. However, an intelligent AI assistant should be proactive, which is capable of anticipating user intentions directly from continuous visual inputs, such as mobile or desktop screenshots, and offering timely recommendations without explicit user prompting. Transitioning to this proactive paradigm presents significant challenges. Real-world screen activity is rarely linear; it consists of long-horizon trajectories fraught with noisy browsing, meaningless actions, and multithreaded task-switching. To address this gap, we introduce PIRA-Bench (Proactive Intent Recommendation Agent Benchmark), a novel benchmark for evaluating multimodal large language models (MLLMs) on continuous, weakly-supervised visual inputs. Unlike reactive datasets, PIRA-Bench features complex trajectories with multiple interleaved intents and noisy segments with various user profile contexts, challenging agents to detect actionable events while fitting to user preferences. Furthermore, we propose the PIRF baseline, a memory-aware, state-tracking framework that empowers general MLLMs to manage multiple task threads and handle misleading visual inputs. PIRA-Bench serves as an initial step toward robust and proactive GUI-based personal assistants.
