Search papers, labs, and topics across Lattice.
This paper introduces TASTE, an innovative method for generating challenging agent benchmarks by reversing the traditional task construction process. By leveraging an Adaptive Contrastive n-gram model trained on LLM-judged validity signals, TASTE produces tasks that not only increase difficulty but also significantly broaden the range of tool-use combinations. The evaluation reveals that agents performing well on existing benchmarks like 蟿^2-Bench experience substantial performance declines on the newly created 蟿^c-Bench, indicating that high scores may reflect saturation rather than genuine problem-solving capability.
Agents that excel on traditional benchmarks may crumble under the pressure of newly synthesized tasks, revealing the limitations of current evaluation methods.
As agent capabilities advance, existing benchmarks, such as 蟿^2-Bench, are becoming increasingly saturated. Yet constructing new benchmark tasks remains complex, costly, and labor-intensive. Moreover, the standard approach, in which scenarios are first written in natural language and then mapped to tool sequences, captures only a narrow subset of the tool-use patterns agents exercise. In this paper, we address these problems by reversing the task construction process. We propose TASTE: Task Synthesis from Tool Sequence Evolution, an automatic method that generates challenging tasks with broader tool-use coverage. TASTE utilizes an Adaptive Contrastive n-gram model trained on LLM-judged validity signals. This enables sampling valid tool sequences that cover a vast range of tool combinations. TASTE then selects representative sequences from the pool via clustering, instantiates them into complete benchmark tasks, and refines them through iterative difficulty evolution. Using TASTE, we construct 蟿^c-Bench, a challenging extension of the three domains of 蟿^2-Bench. We evaluate 11 agent/user LLM pairs and find that models nearly saturating 蟿^2-Bench suffer severe performance drops on our tasks (e.g., Gemini-3-Flash falls from 0.82!-!0.94 to 0.28!-!0.61). Beyond increasing difficulty, our generated tasks more than double the number of unique tool combinations agents must execute. Our results suggest high scores on existing benchmarks often reflect saturation rather than robust task-solving ability. By automating the generation of difficult, high-coverage benchmarks, TASTE enables continuous, scalable evaluation of future agents.
