Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ClawProBench, a novel trace-aware benchmark designed for evaluating AI agents in stateful runtimes, addressing the inadequacy of traditional benchmarks that focus solely on final answers. By implementing two evaluation tracks—one with live workspace tasks and another with closed-world holdouts—the authors demonstrate that native-runtime tasks significantly underperform compared to workspace-live tasks. The findings reveal that traditional correctness-based rankings can obscure critical weaknesses in agent performance, emphasizing the need for a more nuanced evaluation approach that incorporates process quality and safety considerations.
Traditional benchmarks can mask critical weaknesses in AI agents, as ClawProBench reveals that native-runtime tasks score significantly lower than workspace-live tasks.
Agent benchmarks often evaluate only final answers even when agents run on stateful runtimes. We argue this under-specifies what is being evaluated: the proper unit is a declared model-plus-runtime configuration whose failures can occur in evidence acquisition, runtime routing, safety boundaries, or repeated execution. We present ClawProBench, a trace-aware benchmark for runtime-native agent evaluation instantiated on OpenClaw, a live agent runtime with workspace tools and native surfaces for browsing, memory, messaging, scheduling, skills, and subagents. ClawProBench defines two tracks: a 102-scenario full profile with live workspace and native-runtime routing tasks, and a frozen 68-scenario holdout with closed-world JSON output contracts for robust ranking. Trials are scored from execution traces via a safety-gated formula combining correctness, process quality, and efficiency, preserving failure evidence for audit. Our anonymous artifact includes benchmark definitions, scoring code, manifests and sanitized traces. We evaluate 68 configurations on the full profile and 37 on holdout. The top safety-gated average trace score is 0.7671. Native-runtime tasks underperform workspace-live tasks (0.5238 vs. 0.6415). On holdout, pass@k-any outperforms strict three-trial pass (0.6638 vs. 0.2890), while full-profile and holdout rankings show weak alignment (Spearman 0.1300). Rankings based purely on correctness differ substantially from process-aware, safety-gated and strict-pass views. Final-answer leaderboards may hide native-surface weaknesses, one-off successes and trace-local agent failure modes.