Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces VTC-Bench, a novel benchmark designed to evaluate the performance of large language models (LLMs) based on Validated Task Coverage (VTC), which quantifies the number of distinct useful outputs generated across multiple attempts. By focusing on the diversity and relevance of outputs rather than just their individual quality, the authors reveal that traditional evaluation methods can overlook significant differences in model behavior. The findings indicate that configurations yielding the highest single-output quality do not necessarily correlate with the best task coverage, emphasizing the need for a more nuanced evaluation approach in LLM applications.
Conventional evaluation methods can mislead researchers, as models that excel in single-output quality may not provide the most useful diverse outputs in practice.
Many LLM applications are most useful when they provide several candidate outputs for comparison, validation, or combination. Predominant evaluation settings, however, still focus on individual outputs or reduce multiple samples to a single success or selected answer. This can miss whether the outputs include several genuinely different useful results. We introduce VTC-Bench, a five-domain benchmark for this setting, together with Validated Task Coverage (VTC) as its core evaluation quantity. The benchmark is built from carefully selected real-data tasks where both output quality and task-relevant distinctness can be checked automatically and reproducibly, without model-based judges. VTC measures how many distinct useful results are obtained within $k$ attempts. Across multiple models and inference settings, the benchmark leads to different conclusions from conventional evaluation: configurations that look strongest from single-draw quality are not necessarily those with the best coverage, and simple measures of output variation do not reliably recover task-relevant coverage. These results show that finite candidate sets can be evaluated directly as objects of interest, revealing differences in model behavior that are not apparent from conventional per-output evaluation.
RIKEN