Search papers, labs, and topics across Lattice.
This paper introduces MediaWiki Code2Code Search, a neural retrieval system designed to bridge the lexical gap in code search by focusing on semantic intent rather than surface-level tokens. By indexing over 1.29 million structural entities from MediaWiki repositories, the system achieves significant improvements in retrieval precision, particularly in tasks where traditional lexical methods struggle. The architecture features a split-build design that optimizes performance, resulting in a 96.6% reduction in index size and a median query latency of 1.85 seconds, outperforming the BM25 baseline in a comprehensive evaluation.
Semantic code retrieval can outperform traditional methods by 36% in precision, especially in name-obfuscated tasks where lexical approaches fail.
Code search in large-scale ecosystems is often hindered by the lexical gap between user queries and implementation details, alongside the trade-off between the low latency of traditional Information Retrieval (IR) and the precision of Deep Learning (DL). We present MediaWiki Code2Code Search, a neural retrieval system for semantic code-to-code discovery. By indexing 1.29 million structural entities (functions, types, and templates) across 2,500+ MediaWiki repositories, our system enables retrieval based on computational intent rather than surface tokens. We employ a split-build architecture, decoupling GPU-intensive offline indexing from a CPU-only serving layer; our FAISS IVF-PQ index occupies 168.6 MB: a 96.6\% reduction compared to a flat float32 baseline, and achieves a median query latency of 1.85 seconds on commodity hardware, satisfying the 6 GiB RAM constraint of Wikimedia Toolforge. Our evaluation across a 27-query benchmark demonstrates superior performance over the BM25 baseline, achieving a P@10 of 0.87 compared to 0.64 (0.52 versus 0.34 for strict matching). Gains are most pronounced in name-obfuscated tasks where lexical methods fail. The system is available at https://code2codesearch.toolforge.org under the Apache 2.0 licence and provides an open RESTful API.
