Search papers, labs, and topics across Lattice.
This study introduces the Multi-Agent LLM (MALLM) framework to enhance task performance in Large Language Models (LLMs) by employing various decision protocols鈥攙oting, consensus, and judge mechanisms鈥攚ithin multi-agent systems. The research systematically evaluates these protocols across a diverse range of tasks, revealing that consensus protocols outperform in knowledge-intensive scenarios, while voting and judge mechanisms are more suited for logic-based tasks. Notably, increasing response diversity through independent solution generation significantly enhances decision quality, indicating a nuanced approach to multi-agent collaboration in LLMs.
Consensus decision protocols significantly boost performance in knowledge-intensive tasks, reshaping how we leverage multi-agent systems in LLMs.
Improving the task performance of Large Language Models (LLMs) is essential, yet scaling these models faces significant challenges such as diminishing returns and high costs. Multi-Agent Systems (MAS) offer a promising solution by distributing tasks among specialized agents to improve the overall task performance. This can reduce training costs at the expense of increased test time due to the discussion and decision-making process. The decision protocol is a critical component of MAS because it specifies how multiple agents collaborate to create a final solution. This thesis introduces the Multi-Agent LLM (MALLM) framework, which implements and evaluates various decision protocols, namely voting, consensus, and judge decision mechanisms, to simulate multi-agent discussions for conversational task solving. Unlike previous work that used a single decision protocol or tested them on limited datasets, this study systematically examines their impact on a diverse set of tasks, ranging from knowledge-based datasets (MMLU, MMLU-Pro, GPQA) and logic-based datasets (StrategyQA, MuSR, Math-lvl-5, SQuAD 2.0). The results indicate that consensus protocols excel in knowledge-intensive domains while voting and judge protocols are more effective for logic-based tasks. Increasing response diversity through independent solution generation improves decision quality, while changes in information access during the decision process have minimal impact.
