Search papers, labs, and topics across Lattice.
This paper introduces SABER-Math, an automated benchmark designed to evaluate information retrieval (IR) systems specifically for mathematical tasks, addressing the inadequacies of existing benchmarks that fail to capture nuanced mathematical relevance. By leveraging a dataset of 283K high-school-level math problems, the authors develop a three-step process that includes LLM-generated solution summaries, ontology-based document discovery, and a Swiss-style preference tournament for fine-grained relevance ratings. The findings reveal that while modern embedding models outperform traditional methods, they still struggle with symbol-heavy mathematics, underscoring the necessity for dedicated math-focused IR benchmarks.
Modern embedding models excel in general IR tasks but falter in complex mathematical domains, revealing a critical gap in current evaluation benchmarks.
As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources. However, choosing the right retriever remains difficult, as it is infeasible to directly isolate its effect on downstream performance. On the other hand, existing retrieval-specific benchmarks often fail to capture fine-grained mathematical relevance, penalizing relevant documents. We address this gap by introducing SABER-Math, the first fully automated benchmark for evaluating mathematical IR without expert annotation. Starting from 283K high-school-level math problems with solutions, SABER-Math builds challenging reranking tasks in three steps: (i) first, LLMs extract concise solution summaries and mathematical topics for each problem; (ii) then, per-query relevant documents are discovered using ontology topic-based and lexical solutions-summary-based similarities, and (iii) finally, a Swiss-style LLM preference tournament produces fine-grained relevance ratings for the documents. We evaluate lexical retrievers, specialized mathematical retrieval systems, and recent embedding models. We find that while modern embedding models substantially outperform classical and math-specific baselines, even the strongest systems struggle in symbol-heavy domains like Algebra and Calculus. Importantly, we show that general-purpose IR benchmarks such as MTEB do not reliably predict mathematical performance, especially for recent embedding models, highlighting the need for math-specific retrieval benchmarks.
