Search papers, labs, and topics across Lattice.
The paper introduces the Shopping Reasoning Bench, a comprehensive benchmark designed to evaluate multi-turn conversational shopping assistants on their reasoning capabilities and domain expertise. It comprises 525 missions and 10,863 importance-weighted binary rubrics, developed by retail experts, to assess various reasoning categories essential for effective shopping dialogues. Evaluation of nine models reveals that while they perform adequately on basic tasks, they significantly underperform on more complex criteria, highlighting the need for improved models in this domain.
Current conversational shopping assistants only achieve 57-77% pass rates, revealing a critical gap in their ability to provide expert-level advice in multi-turn dialogues.
Conversational shopping assistants now serve hundreds of millions of customers, yet no existing benchmark jointly evaluates the open-ended multi-turn reasoning, domain expertise, and criterion-level quality that real shopping conversations demand. Shopping reasoning is unique among language model applications. Unlike factual question answering or verifiable code generation, it requires balancing subjective preferences, budget constraints, and cross-product trade-offs across multi-turn dialogue, capabilities absent from previous e-commerce and general-purpose benchmarks. We introduce the Shopping Reasoning Bench, an expert-authored benchmark of 525 missions (232 single-turn, 293 multi-turn) with 10863 importance-weighted binary rubrics authored by retail domain experts. These criteria are organized under a taxonomy of five reasoning categories and fifteen subcategories covering diverse demands such as preference refinement, trade-off analysis, and compatibility assessment. An evaluation of nine models across three families (GPT, Claude, Gemini) shows that pass rates reach only 57--77% overall. On multi-turn missions, all models score 13--29 points lower on optional above-and-beyond criteria than on required ones, and performance degrades 4--18 points as conversations progress. These gaps show that current models handle basic shopping assistance but fall short of expert-level advice, making Shopping Reasoning Bench a challenging testbed for future shopping assistant development.
