Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ExecRetrieval, a benchmark designed to evaluate the functional correctness of code-embedding retrieval systems by incorporating execution-verified buggy variants alongside canonical implementations. By testing 23 dense embedding configurations and BM25 against a dataset of 939 Python tasks, the study reveals that while the top system achieves perfect execution accuracy at rank 10, it struggles significantly at rank 1, often misidentifying buggy variants as correct code. The findings highlight a critical gap in current retrieval methods, emphasizing the need for functional discrimination in code retrieval tasks rather than mere lexical similarity.
Despite achieving high accuracy in retrieving code at rank 10, leading systems miss correct implementations at rank 1 nearly 92% of the time, exposing a significant flaw in current embedding-based retrieval approaches.
Embedding-based code retrieval is a core component of coding agents and retrieval-augmented code generation, where retrieving correct code matters more than retrieving lexically similar code. Existing code-retrieval benchmarks do not plant controlled, execution-verified single-edit variants of each query's canonical implementation in the search pool, leaving the question of whether embeddings can functionally discriminate correct from near-clone-but-incorrect code unanswered in a retrieval setting. Resolving this requires a benchmark whose search pool itself contains the relevant counterfactuals -- execution-verified buggy variants near-identical to each canonical -- so that a retriever's rank ordering can be directly tested for functional discrimination rather than topical or identity overlap. We introduce ExecRetrieval, 939 Python tasks each paired with one execution-verified canonical implementation and up to four execution-verified buggy distractors, each generated by a mechanical mutation making a single targeted edit, and evaluate 23 dense embedding configurations plus BM25 under provider-native invocation with paired McNemar tests and query-level bootstrap intervals. With near-clone counterfactuals in the pool, the top hosted system reaches exec@10 = 1.00 but only exec@1 = 0.331; rank-1 misses are paired buggy variants 91.5-99.4% of the time across the four leading systems, and the canonical scores below at least one of its four paired distractors in 67-78% of queries on the leading systems. The full dataset, execution oracle, embedding matrices, environment snapshot, and pairwise statistical tests are released at the URL in Appendix D.