Search papers, labs, and topics across Lattice.
This paper re-evaluates the METR plot's time horizon estimation by employing splines and item-response theory to move beyond linear assumptions between human completion time and AI difficulty. The analysis reveals a non-linear relationship where AI difficulty is relatively stable for tasks between 2-30 minutes, but increases more steeply for shorter and longer tasks. The refined time horizon estimates exhibit improved predictive accuracy via proper scoring rules and are accompanied by diagnostic plots to assess construct validity, advocating for their joint interpretation.
A 10x increase in AI time horizon is significantly easier for AI development in the 2-30 minute task range than for tasks outside this sweet spot.
METR's 50\% time horizon measures the human completion time of software tasks that an AI solves with 50\% probability, allowing AI capabilities to be expressed in interpretable units. On 228 tasks and 26 AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the AI difficulty of a task depends linearly on the log of human time. Our fitted spline can be interpreted as a function that \emph{converts} human time to AI difficulty; it is nearly flat in a region from 2--30 min but close to linear elsewhere. Hence, a time-horizon jump from 3 min to 30 min is much easier than one from 30 min to 5 hours despite the same multiplier of $10 \times$. Overall, we contribute time-horizon point estimates that perform better under a cross-validated suite of proper scoring rules, as well as diagnostic plots for assessing time horizons'construct validity. We suggest that time horizons be interpreted together with the diagnostic plots, especially as new time-horizon-based benchmarks are proposed or existing ones grow to include longer tasks.
