Search papers, labs, and topics across Lattice.
This paper employs a quantitative ethnographic (QE) approach to diagnose and redesign multi-agent large language model (LLM) systems, focusing on how agent interactions influence reasoning outcomes. By analyzing discourse through Epistemic Network Analysis (ENA) in a five-agent debate system, the authors identified key differences in interaction patterns that led to correct versus incorrect automated essay scoring decisions. The redesigned system, informed by these insights, significantly improved scoring accuracy from 27.78% to 40.28% and aligned incorrect debate patterns with those of correct ones, demonstrating the efficacy of QE in enhancing AI reasoning systems.
Multi-agent LLMs can drastically improve reasoning accuracy by redesigning agent interactions based on quantitative ethnographic insights.
Multi-agent large language model (LLM) systems are designed to improve reasoning by decomposing tasks across multiple agents with specialized functions, but the presence of multiple agents does not inherently guarantee coherent reasoning or outputs that align with task objectives. This paper introduces a quantitative ethnographic (QE) approach for diagnosing and redesigning multi-agent LLM systems based on the discourse produced through agent interactions. We test this approach using automated essay scoring as an example context, applying Epistemic Network Analysis (ENA) to model a five-agent multi-agent debate system and examine differences between debates that produced correct versus incorrect scoring decisions. Results show that, in the initial system, correct scoring decisions were characterized by rubric-grounded justification, agreement, and elaboration. Incorrect scoring decisions, in contrast, were characterized by extended proposition-challenge-response exchanges that were less consistently tied to rubric criteria. We then used the findings to revise the agents' prompts. The revised system improved exact scoring accuracy from 27.78% to 40.28% and shifted the discourse of incorrect debates toward the rubric-grounded pattern of correct ones, making the two nearly indistinguishable. Based on these results, we argue that QE can support a diagnostic-to-redesign loop for AI reasoning by tracing how patterns of agent interaction relate to system performance, informing prompt redesign, and evaluating whether those redesigns change both outcomes and interaction patterns.
