Search papers, labs, and topics across Lattice.
This paper introduces RepoProbe, a benchmark designed to assess repository-level code understanding in Large Language Models (LLMs) through open-ended Q&A based on GitHub Discussions, addressing the limitations of existing benchmarks that rely on bug reports. The authors highlight the issue of Edit Bias, where models generate code modifications without fully comprehending the repository architecture, and propose a Checklist-Based Verification Protocol to enhance evaluation reliability by breaking down answers into verifiable facts. Their findings reveal a significant gap in technical correctness among state-of-the-art LLMs, underscoring the need for more rigorous evaluation methods in software engineering applications.
LLMs exhibit a troubling tendency to prioritize code generation over genuine architectural understanding, revealing a critical gap in their evaluation metrics.
The integration of Large Language Models (LLMs) into software engineering has shifted the focus from function-level generation to repository-scale assistance. However, existing benchmarks largely rely on bug reports from GitHub Issues, which often allow models to bypass genuine understanding via pattern matching on error logs. This misalignment under-measures Edit Bias, which refers to premature generation, where models prematurely propose code modifications instead of understanding the existing repository architecture. Furthermore, current LLM-as-a-Judge scalar scoring suffers from high variance and low interpretability. This work introduces RepoProbe, a novel benchmark for evaluating repository-level code understanding through open-ended Q&A using GitHub Discussions, which focuses on open-ended architectural inquiries rather than defect reporting. To ensure rigorous evaluation, we propose a Checklist-Based Verification Protocol that decomposes answers into atomic, verifiable facts, thereby replacing subjective ratings with objective verification. Our evaluation of state-of-the-art (SOTA) LLMs reveals a persistent gap between high clarity and evidencegrounded technical correctness. It also quantitatively confirms the prevalence of edit bias, in which models prioritize code generation instead of architectural analysis. Finally, we demonstrate that our verification protocol significantly improves evaluation reliability compared to traditional evaluations with scalar scoring.
