Search papers, labs, and topics across Lattice.
This study explores the capability of four large language models (LLMs) to extract Architectural Design Decisions (ADDs) from source code commits, addressing the challenge of implicit and unstructured documentation in software systems. By employing zeroshot and fewshot prompting techniques on a dataset of 30 ADDs, the models achieved a BERT-F1 score exceeding 0.81, with fewshot prompting yielding further improvements. Despite these promising results, the generated ADDs often lacked the necessary rationale and were overly focused on implementation details, indicating a need for more architecture-aware LLM systems.
LLMs can achieve over 80% accuracy in extracting architectural insights from code commits, but often miss the critical rationale behind design choices.
Context: Architectural Design Decisions (ADDs) capture the rationale behind the structure and evolution of software systems but are rarely documented explicitly, and are often hidden inside source code commits. Recovering them is important for Architectural Knowledge Management (AKM). Problem: Extracting ADDs from commits is challenging due to their implicit and unstructured nature. Large Language Models (LLMs) have shown strong capabilities in understanding code and text, yet their effectiveness for this task remains underexplored. Study: We present a preliminary study using four LLMs (Gemini 3 Pro, DeepSeek R1, Kimi K2, Qwen3) with zeroshot and fewshot prompting on 30 developer-written ADDs from open-source projects. We score outputs with ROUGE-L, BLEU, METEOR, and BERTScore, and one author manually reviews the Gemini outputs. Results: All models reach a BERT-F1 above 0.81, and fewshot prompting improves alignment (Gemini BERT-F1: 0.828 to 0.847). However, the generated ADDs are often too long, implementation-focused, and miss the rationale behind the decision. This highlights opportunities for architecture-aware LLM systems and automated AKM.
