Search papers, labs, and topics across Lattice.
This paper introduces DuplexWorld, a comprehensive evaluation framework for speech-to-speech voice agents across six practical domains: banking, insurance, travel, healthcare, logistics, and pathfinding. By assessing agents on eleven conversation types across 156 scenarios, the study reveals that even top-performing voice agents exhibit significant deficiencies in agentic performance, conversational engagement, and speech naturalness, with average scores indicating substantial room for enhancement. The findings underscore the need for more nuanced benchmarks that reflect the complexities of real-world conversational interactions beyond simple database queries.
Even the best voice agents struggle with real-world conversational tasks, scoring below 50% in effective assistance across diverse scenarios.
Speech-to-speech (S2S) voice agents are increasingly being incorporated into enterprise for customer care and as daily companions for consumers owing to the ease of the conversational modality over text. However, existing benchmarks fail to holistically evaluate voice agents along axes that really matter and are shaped as tests of agentic tool calling against a database. We believe they fail to adequately account for the diversity of conversational dialogue that mundane activities introduce and further, never test how faithfully an agent can assist on tasks that move beyond database manipulation. To tackle this DuplexWorld introduces six worlds where voice agents are especially useful: banking, insurance, travel, healthcare and logistics, and Pathfinding. Agents are evaluated on eleven different types of conversations across 156 scenarios (350+ hours of conversation), each testing conversational and analytical capability to varying degrees. Through extensive evaluation comprising agentic, conversational and speech-naturalness metrics, we show that even the best voice agents leave substantial room for improvement on all 3 axes (Pass@1: 0.490, turn-taking: 0.653, DNSMOS: 3.378). We perform extensive analysis on agentic v conversational performance, world- and conversation type-wise performance, failure modes exploring the explore v exploit lens for Pathfinding conversations and voice agent reliability over all six worlds.
