Search papers, labs, and topics across Lattice.
The paper introduces CONCUR, a new benchmark designed to evaluate LLMs' ability to generate correct concurrent code, addressing the limitations of existing benchmarks focused on sequential code. CONCUR comprises 115 problems, including 43 base concurrency problems and 72 mutant variants, derived from a standard concurrency textbook. Evaluation of several LLMs using CONCUR reveals shortcomings in current models' ability to handle the complexities and unique bug types (e.g., deadlocks, race conditions) inherent in concurrent programming.
Current LLMs struggle with concurrent code generation, as evidenced by a new benchmark, CONCUR, exposing their inability to handle deadlocks and race conditions.
Leveraging Large Language Models (LLMs) for code generation has increasingly emerged as a common practice in the domain of software engineering. Relevant benchmarks have been established to evaluate the code generation capabilities of LLMs. However, existing benchmarks focus primarily on sequential code, lacking the ability to effectively evaluate LLMs on concurrent code generation. Compared to sequential code, concurrent code exhibits greater complexity and possesses unique types of bugs, such as deadlocks and race conditions, that do not occur in sequential code. Therefore, a benchmark for evaluating sequential code generation cannot be useful for evaluating concurrent code generation with LLMs. To address this gap, we designed a benchmark CONCUR specifically aimed at evaluating the capability of LLMs to generate concurrent code. CONCUR consists of a base set of 43 concurrency problems derived from a standard concurrency textbook, together with 72 validated mutant variants, resulting in 115 total problems. The base problems serve as the semantic core of the benchmark, while the mutants expand linguistic and structural diversity. We conducted an evaluation of a range of LLMs on CONCUR, highlighting limitations of current models. Overall, our work provides a novel direction for evaluating the capability of LLMs to generate code with focus on concurrency.
