Search papers, labs, and topics across Lattice.
This paper introduces MAPLE, a benchmark designed to evaluate multi-aspect retrieval of scientific papers, addressing the limitation of existing benchmarks that focus solely on individual query-paper relevance. By incorporating 2,095 queries that reflect various facets of papers鈥攕uch as motivation, methods, and findings鈥擬APLE enables a more comprehensive assessment of retrieval systems. Experimental results reveal a significant disparity in retrieval performance, with the best model achieving 98.1% for any single aspect but only 15.7% for all aspects combined, highlighting the challenges in multi-aspect retrieval.
Retrieving scientific papers from multiple aspects reveals a staggering performance gap, with top models excelling in single queries but faltering in comprehensive evaluations.
Scientific papers contain multiple searchable facets such as background, methods. However, many paper retrieval benchmarks merely evaluate individual query-paper relevance, while overlooking other facets of the same paper. To bridge this gap, we introduce MAPLE, an expert-validated benchmark for multi-aspect, full-paper retrieval that evaluates whether retrievers can consistently recover the same paper from queries targeting its motivation, method, and experimental findings. MAPLE contains 2,095 queries about recent ML and NLP papers, grounded in both textual and multimodal content. We further propose MAPLE-Synth, a retrieval-based in-context learning pipeline that leverages OpenReview discussions and human-written query exemplars to generate realistic queries reflecting researchers' interests in different aspects of a paper. Our expert validation shows that these queries are comparable in realism to human-written queries and highly relevant to the target papers. Experiments across lexical, scientific-domain, general-purpose text, and multimodal retrievers reveal a substantial gap between retrieving a paper from any one aspect and retrieving it from all aspects: the strongest model achieves 98.1% AnyAspect@20 but only 15.7% AllAspect@20. Experiment/result queries and table-referenced queries are particularly difficult across retrievers. Although multi-chunk aggregation improves multi-aspect paper retrieval, considerable failures persist. MAPLE provides a testbed for evaluating and developing retrievers that represent scientific papers more comprehensively.
