Search papers, labs, and topics across Lattice.
The paper introduces CL4SE, a novel benchmark for evaluating context learning in software engineering tasks, featuring a taxonomy of four SE-specific context types mapped to code generation, summarization, review, and patch assessment. They constructed a dataset of over 13,000 samples from open-source projects and evaluated five LLMs, demonstrating an average performance improvement of 24.7% through context learning. The benchmark and empirical results offer insights into task-specific context design and provide a standardized evaluation framework for SE context learning.
Forget finetuning: carefully chosen context can boost LLM performance on software engineering tasks by up to 33%, and CL4SE provides the benchmark to prove it.
Context engineering has emerged as a pivotal paradigm for unlocking the potential of Large Language Models (LLMs) in Software Engineering (SE) tasks, enabling performance gains at test time without model fine-tuning. Despite its success, existing research lacks a systematic taxonomy of SE-specific context types and a dedicated benchmark to quantify the heterogeneous effects of different contexts across core SE workflows. To address this gap, we propose CL4SE (Context Learning for Software Engineering), a comprehensive benchmark featuring a fine-grained taxonomy of four SE-oriented context types (interpretable examples, project-specific context, procedural decision-making context, and positive&negative context), each mapped to a representative task (code generation, code summarization, code review, and patch correctness assessment). We construct high-quality datasets comprising over 13,000 samples from more than 30 open-source projects and evaluate five mainstream LLMs across nine metrics. Extensive experiments demonstrate that context learning yields an average performance improvement of 24.7% across all tasks. Specifically, procedural context boosts code review performance by up to 33% (Qwen3-Max), mixed positive-negative context improves patch assessment by 30% (DeepSeek-V3), project-specific context increases code summarization BLEU by 14.78% (GPT-Oss-120B), and interpretable examples enhance code generation PASS@1 by 5.72% (DeepSeek-V3). CL4SE establishes the first standardized evaluation framework for SE context learning, provides actionable empirical insights into task-specific context design, and releases a large-scale dataset to facilitate reproducible research in this domain.
