Search papers, labs, and topics across Lattice.
The paper introduces ComboShoppingBench, a benchmark designed to evaluate LLM agents on the complex task of constructing budget-constrained shopping baskets that incorporate coupons and item compatibility. This benchmark addresses the limitations of traditional evaluation metrics by combining semantic assessments with deterministic checks for feasibility and compliance, ensuring that generated baskets meet real-world shopping requirements. Experiments reveal that even advanced LLMs face significant challenges in this domain, indicating a critical need for improved models capable of handling the intricacies of combo shopping.
Even advanced LLMs struggle with budget-constrained combo shopping, revealing a significant gap in their ability to handle real-world constraints effectively.
Real-world shopping often requires constructing a basket of complementary items rather than retrieving a single product. Such combo-shopping tasks arise in device setup, meal preparation, event planning, and group takeout ordering, requiring joint reasoning about item compatibility, availability, store-level requirements, delivery fees, coupons, and budgets. Evaluation is challenging because multiple baskets may satisfy the same request, making exact-match metrics unsuitable, whereas semantic evaluation alone cannot detect infeasible orders, invalid coupon combinations, or incorrect payments. We introduce ComboShoppingBench, an agentic shopping benchmark for open-ended yet verifiable basket construction in a simulated commerce and takeout environment. During task synthesis, an exploration agent constructs a feasible and semantically coherent basket of purchasable products; this witness guides the generation of coupons, budget constraints, user queries, and aligned evaluation rubrics. During evaluation, LLM judges assess semantic satisfaction, response quality, and claim faithfulness, while deterministic validation checks product-ID validity, budget compliance, and coupon optimality. Experiments with diverse LLM agents demonstrate that even strong agents struggle on ComboShoppingBench, highlighting substantial room for improvement in reliable, constraint-aware combo shopping.
