Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces AEMA, a novel evaluation framework for multi-agent LLM systems designed to address limitations in existing single-response scoring methods by enabling process-aware, auditable, and multi-step evaluations. AEMA enhances evaluation stability and human alignment through planning, execution, and aggregation of evaluations across diverse agentic workflows, all under human oversight. Experiments using realistic business scenarios demonstrate AEMA's ability to provide a transparent and reproducible pathway for responsible evaluation, improving upon single LLM-as-a-Judge approaches.
Current LLM-as-a-judge evaluation methods for multi-agent systems lack the stability and human alignment needed for enterprise deployment, a gap AEMA addresses with process-aware, auditable evaluations.
Evaluating large language model (LLM)-based multi-agent systems remains a critical challenge, as these systems must exhibit reliable coordination, transparent decision-making, and verifiable performance across evolving tasks. Existing evaluation approaches often limit themselves to single-response scoring or narrow benchmarks, which lack stability, extensibility, and automation when deployed in enterprise settings at multi-agent scale. We present AEMA (Adaptive Evaluation Multi-Agent), a process-aware and auditable framework that plans, executes, and aggregates multi-step evaluations across heterogeneous agentic workflows under human oversight. Compared to a single LLM-as-a-Judge, AEMA achieves greater stability, human alignment, and traceable records that support accountable automation. Our results on enterprise-style agent workflows simulated using realistic business scenarios demonstrate that AEMA provides a transparent and reproducible pathway toward responsible evaluation of LLM-based multi-agent systems. Keywords Agentic AI, Multi-Agent Systems, Trustworthy AI, Verifiable Evaluation, Human Oversight