Search papers, labs, and topics across Lattice.
This study investigates the prediction of task difficulty in agentic benchmarks without relying on costly rollouts, focusing on 17 diverse domains including coding and mathematics. By identifying token-level entropy as a significant predictive signal, the authors reveal that traditional AUC metrics can obscure subpar difficulty estimates. The findings highlight how discrepancies between expected and observed difficulty can uncover critical flaws in environment design, such as contamination and infeasibility.
Token-level entropy emerges as a powerful predictor of task difficulty, revealing hidden flaws in environment design that traditional metrics overlook.
Task difficulty dictates an agent's likelihood of success, and estimating it without rollouts means forecasting this directly from a task description before executing costly simulations in stateful environments. Reliable estimates would therefore allow environment designers to calibrate evaluation benchmarks and construct progressive training curricula. This becomes increasingly important as agents move into long-horizon domains, where empirical trial-and-error is a severe computational bottleneck. Prior work on early prediction is limited to static tasks or isolated coding environments, often relying on narrow features and inaccurate evaluation metrics. We study \textit{ex ante} difficulty prediction across 17 agentic benchmarks spanning coding, mathematics, machine learning, web navigation, function calling, and other domains. We show that AUC can mask poor difficulty estimates, identify token-level entropy as a useful predictive signal, and show how residuals between expected and observed difficulty can expose hidden environment flaws such as contamination and infeasibility.
