Search papers, labs, and topics across Lattice.
This paper introduces Thinkingbox, a novel sandbox designed for evaluating agents in stateful business workflows, emphasizing the need for agents to manage multi-turn interactions and maintain persistent states effectively. The research highlights that while existing models can achieve a pass rate of 65.36% on initial attempts, their reliability drops significantly when assessed over multiple trials, with only 25.25% success at the 20th attempt. The findings underscore the inadequacy of traditional evaluation metrics, revealing a substantial gap between achieving a single successful outcome and ensuring consistent task completion in complex environments.
Achieving a single successful task completion in stateful workflows doesn't guarantee reliability, with many agents failing to maintain consistent performance across multiple attempts.
Recent agent benchmarks increasingly ground evaluation in executable environments, from code repair to web navigation, app APIs, and function calling. Yet completing consequential work beyond code requires more than producing a plausible response or valid tool call: agents must gather missing information over multiple turns, follow domain policies, coordinate dependent tools, and realize the correct persistent state transition without collateral effects. In this paper, we introduce Thinkingbox, a sandbox for tool-agent-user interaction that provides isolated MCP-compatible tool sessions, complete execution traces, and outcome evaluation over terminal backend state. Built on this sandbox, Thinkingbox-bench contains 507 policy-conditioned workflows across numerous scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support. Each attempt is evaluated by task-specific executable checks that accept valid trajectories while rejecting wrong, missing, or extra effects; designated tasks additionally check required properties of the final response. Across proprietary and open-weight models, the strongest achieves 65.36% pass@1, but only 25.25% pass^20. Moreover, many failed trials show clean termination and valid state-changing actions, showing that response or tool-call-level signals are not clear proxies for end-to-end task completion. Thinkingbox-bench reveals a large gap between occasionally finding a successful trajectory and reliably completing stateful business tasks. We release both Thinkingbox and Thinkingbox-Bench: https://github.com/microsoft/thinkingbox
Microsoft Research