Search papers, labs, and topics across Lattice.
The paper introduces CAKE, a new benchmark to evaluate LLMs' understanding of cloud-native software architecture, comprising 188 expert-validated questions across Bloom's taxonomy and cloud topics. They evaluated 22 model configurations (0.5B-70B parameters) using majority voting for multiple-choice and LLM-as-a-judge scoring for free-response questions, revealing performance plateaus in MCQs but continued scaling in free-responses. Their findings highlight the importance of evaluation format and the nuanced impact of reasoning and tool augmentation on architectural knowledge assessment.
LLMs ace cloud architecture MCQs, but free-response answers reveal a deeper, still-scaling understanding, suggesting current benchmarks may overestimate true architectural knowledge.
In today's software architecture, large language models (LLMs) serve as software architecture co-pilots. However, no benchmark currently exists to evaluate large language models'actual understanding of cloud-native software architecture. For this reason we present a benchmark called CAKE, which consists of 188 expert-validated questions covering four cognitive levels of Bloom's revised taxonomy -- recall, analyze, design, and implement -- and five cloud-native topics. Evaluation is conducted on 22 model configurations (0.5B--70B parameters) across four LLM families, using three-run majority voting for multiple-choice questions (MCQs) and LLM-as-a-judge scoring for free-responses (FR). Based on this evaluation, four notable findings were identified. First, MCQ accuracy plateaus above 3B parameters, with the best model reaching 99.2\%. Second, free-response scores scale steadily across all cognitive levels. Third, the two formats capture different facets of knowledge, as the MCQ accuracy approaches a ceiling while free-responses continue to differentiate models. Finally, reasoning augmentation (+think) improves free-response quality, while tool augmentation (+tool) degrades performance for small models. These results suggest that the evaluation format fundamentally shapes how we measure architectural knowledge in LLMs.
