Search papers, labs, and topics across Lattice.
The paper introduces ProVoice-Bench, a new benchmark designed to evaluate the proactivity of voice agents across four novel tasks requiring proactive intervention and monitoring. They use a multi-stage data synthesis pipeline to generate 1,182 high-quality samples. Evaluation of current multimodal LLMs on ProVoice-Bench reveals significant performance gaps, especially in over-triggering and reasoning, highlighting areas for improvement in proactive agent design.
Today's best multimodal LLMs are surprisingly bad at knowing *when* to speak up, revealing a critical flaw in the shift towards proactive voice agents.
Recent advancements in LLM agents are gradually shifting from reactive, text-based paradigms toward proactive, multimodal interaction. However, existing benchmarks primarily focus on reactive responses, overlooking the complexities of proactive intervention and monitoring. To bridge this gap, we introduce ProVoice-Bench, the first evaluation framework specifically designed for proactive voice agents, featuring four novel tasks. By leveraging a multi-stage data synthesis pipeline, we curate 1,182 high-quality samples for rigorous testing. Our evaluation of state-of-the-art Multimodal LLMs reveals a significant performance gap, particularly regarding over-triggering and reasoning capabilities. These findings highlight the limitations of current models and offer a roadmap for developing more natural, context-aware proactive agents.
