Search papers, labs, and topics across Lattice.
OrchBench introduces a simulation-based benchmark for evaluating multi-agent orchestration plans in isolation, addressing the limitations of traditional end-to-end evaluations that conflate orchestration quality with agent capabilities and environmental factors. By constructing directed acyclic graphs (DAGs) from real-world tasks, OrchBench allows for controlled assessments of task dependencies and agent assignments, resulting in a deterministic evaluation of orchestration plans. The benchmark demonstrates strong correlation with execution quality in real scenarios, achieving significant reductions in token and time costs while revealing critical insights about the importance of information retention over mere agent count.
OrchBench reveals that preserving task-critical information is more crucial than simply increasing agent numbers, challenging conventional wisdom in multi-agent system design.
Complex tasks often decompose into parallelizable yet interdependent subtasks, making orchestration critical to the performance of multi-agent systems (MAS). Existing evaluations typically rely on end-to-end execution, which conflates orchestration-plan quality with worker capabilities, tool reliability, and environmental noise. Moreover, the time and token costs of real execution grow rapidly with workflow scale, making systematic evaluation expensive. We present OrchBench, a simulation-based benchmark for evaluating multi-agent orchestration plans in isolation. Starting from real-world tasks, OrchBench constructs directed acyclic graphs (DAGs) that encode task dependencies, with controlled sizes and degrees of parallelism. Given a DAG, a per-agent context limit, and an agent budget, the evaluated planner assigns subtasks to agents and specifies cross-agent information transfers and their retention ratios. A deterministic simulator evaluates the resulting plan without invoking worker agents and returns interpretable measures of result quality, makespan, and token cost. The simulated scores produced by OrchBench correlate strongly with quality scores from Claude Code executions, achieving a Pearson correlation of \(r=0.816\), while requiring only \(1.3\%\) of the tokens and \(10.3\%\) of the wall-clock time. Across diverse planners and workflow scales, we find that preserving task-critical information is more important than simply increasing the number of agents, and the benefits of parallelism diminish as coordination failures accumulate. These results establish OrchBench as an efficient and interpretable benchmark for comparing and diagnosing multi-agent orchestration plans.
