Search papers, labs, and topics across Lattice.
AutoIndex introduces a novel framework for learning representation programs that transform raw documents into optimized representations for retrieval systems. By employing a validation-guided program search, the framework iteratively diagnoses failures and synthesizes improvements, leading to significant enhancements in retrieval quality. Evaluated on the CRUMB benchmark, AutoIndex achieves average gains of +8.4% in Recall@100 and +8.3% in nDCG@10 compared to a static BM25 baseline, with maximum improvements of +30.5% and +43.6%, respectively.
Learning dynamic representation programs can boost retrieval performance by over 30% compared to traditional static methods.
We present AutoIndex, a framework for learning representation programs: executable transformations that map raw documents into the representations exposed to a retrieval system. Rather than tuning retrievers, rerankers, or a small set of preprocessing hyperparameters, AutoIndex searches over programs that slice, enrich, normalize, reweight, or reorganize documents before indexing. At each iteration, AutoIndex performs validation-guided program search, in which agents diagnose failures of the current program and synthesize candidate updates, retaining only updates that improve retrieval quality under the resulting index. We evaluate AutoIndex on CRUMB, a benchmark of heterogeneous retrieval tasks, with BM25 held fixed across all experiments. The learned programs improve recall over a static full-document BM25 baseline on all 8 tasks, with average gains of +8.4% in Recall@100 and +8.3% in nDCG@10, and largest gains of +30.5% in Recall@100 and +43.6% in nDCG@10. These results suggest that document representation should not be treated as a fixed preprocessing choice made before retrieval begins, but as an explicit optimization target. Code to reproduce our results is available at https://github.com/auto-index/autoindex.
