Search papers, labs, and topics across Lattice.
This paper introduces LLVM-Bench, a comprehensive benchmark designed to evaluate large language models (LLMs) on real-world LLVM compiler issue resolution tasks, comprising 423 validated instances. The study reveals that current LLMs struggle with significant failure modes such as patch invalidity and build failures, highlighting the limitations of existing techniques in this complex domain. By leveraging a novel ensemble approach called LLVM-Ens, which integrates diverse patch generation methods, the authors achieve a notable improvement in resolution rates, reaching up to 21.99%.
Current LLMs falter in resolving LLVM compiler issues, but a new ensemble method boosts resolution rates by nearly 22%.
LLVM is a widely used compiler infrastructure whose scale and complexity make issue resolution labor-intensive and challenging. Although large language models (LLMs) have recently achieved remarkable success in issue resolution, their effectiveness on complex system-level LLVM compiler remains largely unexplored. To address this gap, we introduce LLVM-Bench, the first large-scale benchmark for LLVM issue resolution, containing 423 real-world, validated tasks collected from the LLVM project. We further develop LLVM-Gym, a scalable evaluation platform that automates issue reproduction, patch application, compiler building, and test execution. Using LLVM-Bench and LLVM-Gym, we conduct a comprehensive study of four representative LLMs, six retrieval configurations, and three agents. Our results show that current LLM-based issue resolution techniques remain limited on LLVM-Bench, with patch invalidity and build failures as the dominant failure modes. We further reveal a strong complementarity among different LLMs and agents, motivating LLVM-Ens, a lightweight ensemble approach that expands the patch space through integrating the patches generated by diverse techniques, filters incorrect and redundant candidates, and identifies the most promising solution. Our results show that LLVM-Ens achieves a resolution rate of up to 21.99%, further improving LLVM issue resolution.
