Search papers, labs, and topics across Lattice.
This paper introduces CEO-Bench, a benchmark designed to evaluate the performance of language model agents in complex, long-term tasks by simulating the operation of a startup over 500 days. The benchmark assesses agents on their ability to navigate uncertainty, adapt to changes, and coordinate multiple business functions, revealing that while some advanced models like Claude Opus 4.8 and GPT-5.5 can maintain a positive balance, they struggle to achieve consistent profitability. The findings highlight significant limitations in current state-of-the-art models when faced with real-world complexities, underscoring the need for further advancements in AI capabilities for sustained decision-making.
Only the most advanced language models can keep a startup afloat in a simulated environment, yet they still fail to achieve consistent profitability.
Language model agents are becoming proficient executors at isolated, short-horizon tasks such as software engineering and customer service. Yet real-world challenges require a combination of sophisticated skills that remain largely untested in agents: (1) navigating long horizons amid uncertainty; (2) acquiring information in noisy environments; (3) adapting to a changing world; (4) orchestrating multiple moving parts toward a coherent goal. We introduce CEO-Bench, which evaluates these capabilities together by simulating a representative real-world task: operating a startup for 500 days. An agent manages pricing, marketing, budgeting, and many other aspects of a fictional company through a programmable Python interface, operating in the same environment and facing the same challenges as a human CEO. Success demands analyzing noisy, interconnected business databases, translating signals into sound strategy, and coordinating many decisions with programming. The strongest agents write sophisticated code that simulates customer cohorts to forecast future cash and mines negotiation history to uncover hidden customer preferences. Even so, most state-of-the-art models struggle in this environment. Only Claude Opus 4.8 and GPT-5.5 finish above the $1M starting balance, and neither consistently turns a profit. CEO-Bench takes a first step toward measuring the intelligence required to drive sustained, adaptive progress over time.
