Search papers, labs, and topics across Lattice.
The paper introduces RetailBench, a new benchmark for evaluating LLM agents in long-horizon, realistic retail environments with stochastic demand. They propose the Evolving Strategy & Execution framework to decouple high-level strategic reasoning from low-level action execution, enabling adaptive strategy revision. Experiments with eight LLMs demonstrate that the framework improves stability and efficiency, but performance still degrades significantly with increasing task complexity, highlighting limitations of current LLMs in such scenarios.
LLM agents struggle to maintain coherent decision-making in realistic retail environments over long horizons, even with a novel framework for adaptive strategy evolution.
Large Language Model (LLM)-based agents have achieved notable success on short-horizon and highly structured tasks. However, their ability to maintain coherent decision-making over long horizons in realistic and dynamic environments remains an open challenge. We introduce RetailBench, a high-fidelity benchmark designed to evaluate long-horizon autonomous decision-making in realistic commercial scenarios, where agents must operate under stochastic demand and evolving external conditions. We further propose the Evolving Strategy & Execution framework, which separates high-level strategic reasoning from low-level action execution. This design enables adaptive and interpretable strategy evolution over time. It is particularly important for long-horizon tasks, where non-stationary environments and error accumulation require strategies to be revised at a different temporal scale than action execution. Experiments on eight state-of-the-art LLMs across progressively challenging environments show that our framework improves operational stability and efficiency compared to other baselines. However, performance degrades substantially as task complexity increases, revealing fundamental limitations in current LLMs for long-horizon, multi-factor decision-making.
Stanford HAI