Search papers, labs, and topics across Lattice.
This paper introduces PeakBench, a benchmark designed to evaluate the performance of LLM agents in executing multi-tool workflows under resource constraints. The study reveals that while strong logical planning is crucial, it does not guarantee safe or efficient execution, highlighting a significant gap in existing benchmarks that primarily focus on tool selection and argument generation. By implementing a two-part evaluation framework that separates logical planning from physical scheduling, PeakBench demonstrates that providing resource information can significantly enhance resource utilization and reduce overflow incidents.
Strong logical planning in LLM agents doesn't ensure safe execution鈥擯eakBench reveals critical flaws in existing benchmarks that could lead to resource overflows.
LLM agents increasingly solve tasks by invoking multiple tools, where parallel execution is essential for low latency but difficult to manage safely. Existing agent benchmarks primarily evaluate tool selection, argument generation, and end-to-end success under mostly serial execution, largely overlooking valid parallelization and resource-constrained scheduling. This missing scheduling dimension creates a practical failure mode: serial execution is safe but slow, while resource-agnostic parallel execution is fast but prone to avoidable resource overflows. To address this gap, we introduce PeakBench, a benchmark of executable multi-tool workflows with execution-grounded dependency annotations and measured resource profiles. A central challenge in evaluating such workflows is attribution: failures and inefficiencies may arise from incorrect dependency planning, poor resource-constrained scheduling, or both. PeakBench addresses this challenge with a two-part evaluation framework that disentangles logical planning from physical scheduling, with dedicated metrics for each dimension. Using this framework, we show that strong logical planning does not reliably translate into safe or efficient execution under resource constraints. We further show that exposing resource information can reduce avoidable overflows and improve resource utilization, making PeakBench a useful testbed for diagnosing resource-aware agent behavior. Code is available at https://github.com/Czzzk/Staggering-the-Peaks.
