Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper presents QuickScope, a novel methodology for identifying hard questions within dynamic LLM benchmarks by modifying the COUP Bayesian optimization algorithm for practical use in LLM evaluation. The approach addresses the challenges of evaluating models not just on average performance but on their weaknesses, which is crucial for advancing model robustness. Experimental results demonstrate that QuickScope outperforms standard baselines in sample efficiency and reduces false positives, making it a valuable tool for targeted LLM evaluation.
QuickScope uncovers hard questions in LLM benchmarks more efficiently than traditional methods, revealing model weaknesses that could otherwise go unnoticed.
LLM benchmarks are increasingly dynamic: instead of containing a fixed set of questions, they define templates and parameters that can generate an effectively unlimited number of question variants. This flexibility is valuable, but it makes evaluation expensive -- especially when the goal is not just determining an average score, but reliably identifying a model's weak spots. This paper introduces a new methodology for identifying hard questions in dynamic benchmarks. It leverages COUP, a recent Bayesian optimization algorithm (Graham, Velez & Leyton-Brown, 2026), after introducing several substantive modifications to make the algorithm suitable for practical LLM pipelines. We also wrap it in a tool that supports flexible choices of datasets and utility functions, enabling users to target the kinds of questions they care about (e.g., low-accuracy questions; questions that are unusually hard relative to their measured complexity). In experiments across a range of benchmarks, we show that our method, dubbed $\texttt{QuickScope}$, discovers truly difficult questions more sample efficiently than standard baselines, while also reducing false positives from noisy outcomes.