Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To streamline software repository analysis without cumbersome multi-stage retrieval pipelines, the authors design a tool-augmented GPT-4 system that translates natural language queries into real-time API calls across issues, pull requests, and commits. Bypassing standard vector-embedding and retrieval-augmented generation (RAG) architectures, the approach focuses on structured parameter extraction and prompt-driven tool routing. Evaluation across compound and domain-specific query categories shows that direct tool orchestration reliably yields accurate, actionable repository intelligence for both technical and non-technical stakeholders.
Vector databases and dense retrievers are often overkill for software intelligence: prompt-driven tool selection over live repository APIs delivers accurate, real-time analytics with significantly less architectural complexity.
Software repositories contain vast amounts of data on code contributions, bug reports, and project activities, yet this information remains challenging for non-technical stakeholders and developers to access due to limited expertise in querying repositories. To address this, we introduce a novel chatbot architecture leveraging OpenAI's GPT-4 model for automated extraction and analysis of repository data. In contrast, our architecture takes a structured path first by parsing the user's query to extract relevant parameters, then selecting the correct tool to employ based on that analysis, and finally invoking the GPT-4 model to create a highly detailed response. In contrast to previous work based on multi-component systems with embedding models and document retrievers, our architecture inverts the process by relying on prompt engineering and tool selection to fit with the query intent. To validate our approach, we conducted experiments on various question types, including Issues, Pull Requests, Commits, Compound Questions, and General Repository Information, evaluating our target prompts' ability to improve the accuracy of responses from the model. Beyond demonstrating the utility of this architecture to a diverse set of users, our findings suggest that this architecture can make repository data more accessible to technical and non-technical audiences through the production of actionable insights.