Search papers, labs, and topics across Lattice.
This paper introduces a comprehensive vocabulary for multi-agent automated research systems, aimed at clarifying design choices and facilitating comparison among different systems. By defining key components such as agent roles, operational capabilities, communication methods, and evaluation processes, the framework allows for a structured approach to understanding the stochastic nature of these systems and their outputs. The authors illustrate the vocabulary's applicability through various autoresearch systems, highlighting its utility in addressing structural design questions and improving evaluative metrics.
A new vocabulary for multi-agent automated research systems reveals how design choices can significantly impact the quality and novelty of generated outputs.
We introduce a vocabulary for automated research systems built from one or more agents to make their design choices easier to describe and compare. The vocabulary specifies 1) who the agents are, 2) what operations are available in the system, 3) who may invoke them, 4) how agents communicate, 5) what information is visible within and across runs, 6) how the next action is chosen, 7) how a run begins, and 8) how outputs are evaluated. A trajectory records one run from the input task to the returned artifact. Because agents, operations, and initialization may be stochastic, repeated runs on the same task induce a distribution over trajectories rather than a single behavior. Our vocabulary turns structural design questions, such as when agents should communicate, gain or lose a capability, or carry information across runs, into testable choices. It also makes the evaluator a component of the system, since reported gains depend on how closely the proxy score matches true quality. That separation also splits the vague complaint that these systems lack taste into two failures with different solutions. Generative taste is the rate at which a system proposes novel trajectories before any score is observed, and evaluative taste is the gap between the proxy score and the quality it should match. We instantiate the vocabulary on recent autoresearch systems to illustrate that it covers designs that differ widely in structure.
