Search papers, labs, and topics across Lattice.
AutoIndex introduces a novel framework for learning representation programs that transform raw documents into optimized representations for retrieval systems. By employing a validation-guided program search, AutoIndex iteratively diagnoses and updates document processing strategies, significantly enhancing retrieval performance. Evaluated on the CRUMB benchmark, the framework achieved notable improvements in recall and nDCG, demonstrating that document representation should be dynamically optimized rather than statically defined.
AutoIndex achieves up to 30.5% improvement in recall by dynamically optimizing document representations for retrieval tasks.
We present AutoIndex, a framework for learning representation programs: executable transformations that map raw documents into the representations exposed to a retrieval system. Rather than tuning retrievers, rerankers, or a small set of preprocessing hyperparameters, AutoIndex searches over programs that slice, enrich, normalize, reweight, or reorganize documents before indexing. At each iteration, AutoIndex performs validation-guided program search, in which agents diagnose failures of the current program and synthesize candidate updates, retaining only updates that improve retrieval quality under the resulting index. We evaluate AutoIndex on CRUMB, a benchmark of heterogeneous retrieval tasks, with BM25 held fixed across all experiments. The learned programs improve recall over a static full-document BM25 baseline on all 8 tasks, with average gains of +8.4% in Recall@100 and +8.3% in nDCG@10, and largest gains of +30.5% in Recall@100 and +43.6% in nDCG@10. These results suggest that document representation should not be treated as a fixed preprocessing choice made before retrieval begins, but as an explicit optimization target. Code to reproduce our results is available at https://github.com/auto-index/autoindex.
