Search papers, labs, and topics across Lattice.
The authors introduce $\pi$-Bench, a new benchmark designed to evaluate the proactive capabilities of personal assistant agents in long-horizon, multi-turn interactions. The benchmark consists of 100 tasks across 5 user personas, incorporating hidden user intents, inter-task dependencies, and cross-session continuity to better simulate real-world scenarios. Experiments using $\pi$-Bench reveal that proactive assistance remains a significant challenge for current agents, highlighting a distinction between task completion and true proactivity, while also demonstrating the benefits of leveraging prior interactions for improved intent resolution.
Current personal assistant agents struggle to anticipate and act on unstated user needs in long, complex workflows, revealing a critical gap between task completion and genuine proactivity.
The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work. A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated. However, existing benchmarks rarely evaluate whether agents can identify and act on such hidden intents before they are explicitly stated, especially in sustained multi-turn interactions where user needs emerge gradually. To address this gap, we introduce $\pi$-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas. By incorporating hidden user intents, inter-task dependencies, and cross-session continuity, $\pi$-Bench evaluates agents'ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use. Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.
