Harry Coppock

Agents in CivBench under-monitor strategic states and frequently fail to act on their own planning commitments, revealing critical gaps in tool-mediated decision-making.

AI agents can handle the engineering aspects of research but fall short in addressing the core research questions, leading to outright rejections from experts.

Fixed-budget evaluations can significantly underestimate the capabilities of advanced language models, with larger token budgets unlocking their true potential across diverse tasks.