Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper introduces R2Eval, a new benchmark for evaluating LLMs' code reasoning abilities using 135 problems extracted from 10 real-world Python projects. R2Eval addresses limitations of existing benchmarks by serializing complex and custom data types, thus preserving the intricate dependencies found in practical software development. Experiments using R2Eval reveal that current LLMs struggle to reason effectively with real-world code contexts, highlighting a gap in their practical generalizability.
LLMs that ace synthetic code benchmarks still stumble when faced with the messy reality of real-world Python projects.
Code reasoning tasks are increasingly crucial to evaluating large language models (LLMs). Yet most existing benchmarks rely on simplistic, LLM-generated snippets or human-written solutions to code challenges and often restrict inputs and outputs to primitive types, failing to reflect the structure and dependencies of real-world projects. These simplifications limit their ability to measure practical generalizability. We present R2Eval1, a benchmark of 135 code reasoning problems drawn from ten widely used Python projects. Unlike prior work, R2Eval serializes compound and custom types, preserving real-world data complexity and enabling a more realistic assessment of LLMs.