Search papers, labs, and topics across Lattice.
The authors introduce CLARC, a new C/C++ code search benchmark designed to stress-test the robustness of retrieval models beyond lexical cues. CLARC uses an automated pipeline to generate query-code pairs from real-world GitHub repositories, incorporating LLM-generated natural language queries validated through human scoring. Evaluation of six state-of-the-art models on CLARC, under identifier anonymization and compilation to Assembly/WebAssembly, reveals significant performance drops, indicating a reliance on lexical features rather than semantic understanding.
Code search models still lean heavily on lexical features, failing to grasp code semantics, as revealed by a new C/C++ benchmark that anonymizes identifiers and compiles code to low-level languages.
Efficient code retrieval is critical for developer productivity, yet existing benchmarks largely focus on Python and rarely stress-test robustness beyond superficial lexical cues. To address the gap, we introduce an automated pipeline for code search datasets and present CLARC, a C/C++ benchmark built from real-world GitHub repositories. CLARC contains 1,245 query-code pairs for evaluation and 5,472 pairs for training. The benchmark incorporates LLM-generated natural language queries validated through rigorous human scoring and hypothesis testing. To analyze contextual requirements effectively, our pipeline starts by ensuring code compilability. It then categorizes code snippets by dependency complexity, distinguishing whether the code relies on custom-defined types or helper functions. The pipeline also enables CLARC to stress-test retrieval robustness by introducing challenging settings, including identifier anonymization and compilation to low-level languages like Assembly and WebAssembly. Under these conditions, our evaluation of six state-of-the-art models reveals sharp drops in retrieval effectiveness. The experimental results highlight the models' persistent reliance on lexical features rather than code semantic understanding. Our dataset is publicly available at https://huggingface.co/datasets/ClarcTeam/CLARC.
