Search papers, labs, and topics across Lattice.
This paper introduces a new benchmark for evaluating LLM agents in long-term, preference-aware e-commerce tasks, addressing the lack of suitable evaluation datasets. They propose Shopping Companion, a unified framework that jointly optimizes memory retrieval and shopping assistance, enabling end-to-end training with user intervention. A dual-reward reinforcement learning strategy with tool-wise rewards is used to train the framework, and experiments show that Shopping Companion outperforms strong baselines on the new benchmark, even surpassing GPT-4.
Even GPT-4 struggles with long-term preference capture in e-commerce, but a lightweight, jointly-trained LLM agent can beat it.
In e-commerce, LLM agents show promise for shopping tasks such as recommendations, budgeting, and bundle deals, where accurately capturing user preferences from long-term conversations is critical. However, two challenges hinder realizing this potential: (1) the absence of benchmarks for evaluating long-term preference-aware shopping tasks, and (2) the lack of end-to-end optimization due to existing designs that treat preference identification and shopping assistance as separate components. In this paper, we introduce a novel benchmark with a long-term memory setup, spanning two shopping tasks over 1.2 million real-world products, and propose Shopping Companion, a unified framework that jointly tackles memory retrieval and shopping assistance while supporting user intervention. To train such capabilities, we develop a dual-reward reinforcement learning strategy with tool-wise rewards to handle the sparse and discontinuous rewards inherent in multi-turn interactions. Experimental results demonstrate that even state-of-the-art models (such as GPT-5) achieve success rates under 70% on our benchmark, highlighting the significant challenges in this domain. Notably, our lightweight LLM, trained with Shopping Companion, consistently outperforms strong baselines, achieving better preference capture and task performance, which validates the effectiveness of our unified design.
