Search papers, labs, and topics across Lattice.
This paper introduces MerchantBench, a novel benchmarking framework designed to evaluate the long-term coherence of large language model (LLM) agents in e-commerce operations. By simulating a 365-day order-level environment with real product data and interdependent decision-making, the study assesses how well LLMs can maintain purposeful behavior and adapt to evolving circumstances. Results indicate that even the most advanced LLM configurations achieve only 27.3% of the final net assets reached by human participants, highlighting significant limitations in current AI capabilities for sustained decision-making.
LLMs struggle to match human decision-making in e-commerce, achieving only 27.3% of the final net assets in a year-long simulation.
Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3\% of the mean final net assets achieved by human participants.
