Search papers, labs, and topics across Lattice.
This paper introduces the "Business Arena," a benchmarking environment designed to evaluate LLM agents' capabilities in managing a cross-border shop using real-world data from Alibaba.com. By simulating complex business workflows and measuring profit alongside skill-level metrics, the authors reveal significant performance disparities among 15 frontier models, with a ninefold difference in final net worth. The findings indicate that while LLM agents show potential, they still lag behind human-designed strategies, highlighting the complexities of business operations that remain challenging for AI.
LLM agents exhibit a staggering ninefold difference in performance when navigating the complexities of a simulated marketplace, yet still fall short of human-designed strategies.
Running a business is a challenging form of intelligent work. Operators must infer opportunities from partial signals, commit capital under uncertainty, adapt to delayed outcomes in a changing market, and satisfy regulatory obligations before trading legally. Frontier LLM agents can increasingly complete complex workflows, yet business-related capabilities are rarely evaluated in existing agent benchmarks. We introduce \textbf{Business Arena}, a controlled environment where an AI agent runs a cross-border shop, buying from suppliers and selling to buyers over a long horizon. We ground the arena in real Alibaba.com sourcing data and market conditions calibrated from authoritative sources. Delayed and coupled consequences make individual business decisions difficult to judge, but their combined outcome is measurable through profit. Because profit alone cannot explain why an agent succeeds or fails, we compare agents with human-designed strategies to estimate available opportunity, use skill-level metrics to reveal underlying strengths and weaknesses, and trace realized gains and losses to the actions that produced them. We use mechanism ablations to establish that strong results reflect genuine business intelligence rather than neglect or simulator-specific shortcuts. We evaluate 15 frontier models and find a ninefold difference in mean final net worth. Even the best model falls behind human-designed strategies, indicating that business operation remains challenging for LLM agents. Skill-level analysis reveals operating styles, from margin-focused premium sellers to high-turnover wholesalers and customer-service specialists, while action-level attribution identifies the sourcing, pricing, and recovery decisions that create or destroy value. Together, Business Arena takes a first step toward a realistic and trustworthy testbed for evaluating end-to-end business agents.
