Search papers, labs, and topics across Lattice.
This paper introduces CentaurBench, a novel benchmarking framework that evaluates the capabilities of large language models (LLMs) in both augmenting and automating real-world work tasks. By comparing model performance in assisting a lower-capacity worker versus direct automation, the study reveals that the rankings of models in these two contexts are only modestly correlated, with the automation winner often underperforming in augmentation tasks. Notably, in several instances, unaided workers outperform assisted conditions, indicating that reliance on automation metrics alone may misrepresent a model's true effectiveness in collaborative settings.
Automation metrics can mislead model selection, as LLMs that excel in direct output generation often fail to enhance the performance of weaker agents.
Most LLM benchmarks rank models on their ability to automate work tasks. In practice, however, models are often used to assist other (human or LLM) agents. The question that drives model selection is therefore not only which model produces the best output, but which model most improves the work of another (weaker) agent. We introduce a unified framework that evaluates the capability of models to automate and augment another agent's performance. Across seven economically grounded real-world tasks, an assistant model writes assistance text for a standardized lower-capacity worker model, which produces the deliverable. In automation mode, the assistant produces the output directly. Outputs are scored through blind pairwise comparisons by an LLM judge panel with task-specific rubrics, replicated across ten runs. Rankings across the two regimes are only modestly correlated, and the automation winner loses augmentation on five of seven tasks. Assistance is not reliably positive. The unaided worker outranks every assisted condition on three tasks, and only one model's guidance beats no guidance on average. These results suggest that automation ability is an incomplete proxy for assistance quality, motivating benchmarks that evaluate models according to the roles they play in human-AI and multi-agent systems.
BAIR