Search papers, labs, and topics across Lattice.
This paper explores the implementation of multi-agent systems (MAS) powered by large language models (LLMs) in software engineering, highlighting the transition from single-agent applications to collaborative frameworks. Through a mixed-method approach, the authors quantitatively analyzed open-source MAS frameworks and qualitatively evaluated their performance on a common task鈥攕ummarizing README.MD files. The results indicate that while these frameworks cover essential MAS components, they lack advanced features, and there is no significant difference in summarization quality across the evaluated frameworks.
Despite the promise of multi-agent systems in software engineering, key frameworks still lack advanced features and show no significant performance difference in summarization tasks.
The proliferation of Generative Artificial Intelligence (Gen AI) powered by large language models (LLMs) has transformed the software development process, introducing new paradigms for code generation, debugging, testing, and maintenance. While early applications focused on leveraging single, independent LLMs to assist developers with isolated tasks, recent advances have shifted toward multi-agent systems (MAS) that orchestrate multiple LLM-based agents working collaboratively toward common objectives. Despite their promising potential, using MAS encompasses a set of challenges for developers who have to carefully select the right technology, devise proper coordination rules, and design specific roles for the involved agents. In this paper, we provide a comprehensive overview of the existing tools and frameworks for implementing MAS in software engineering. First, we conducted a quantitative analysis of the most relevant open source MAS frameworks by evaluating their documentation, features, and capabilities from the developers'perspective. Second, we performed a qualitative evaluation of a subset of the selected frameworks by implementing a common use case: the summarization of README.MD files. The findings show that the selected frameworks provide a good coverage of fundamental components of MAS, though advanced features such as telemetry of agents are still missing. In addition, the empirical evaluation shows that there is no significant difference in terms of ROUGE scores considering the summarization task. Finally, we provide a set of lessons learned and challenges that can help researchers and practitioners to select a suitable MAS framework according to their needs.
