Search papers, labs, and topics across Lattice.
This paper introduces two novel benchmarks, mamabench and mamaretrieval, specifically designed to evaluate medical retrieval-augmented generation in maternal, neonatal, and reproductive health contexts. By assembling a comprehensive QA set of 25,949 items and a relevance-graded corpus of 3,185 clinical queries, the authors address a significant gap in existing medical benchmarks that inadequately cover the nuanced questions faced by healthcare professionals in these fields. The benchmarks not only enhance the assessment of language models in this critical area but also provide a transparent methodology for evaluating relevance, improving the robustness of AI applications in maternal health.
A groundbreaking benchmark reveals that existing medical QA systems overlook critical maternal and reproductive health queries, paving the way for more effective AI-driven healthcare solutions.
Medical question-answering benchmarks rarely cover the maternal, neonatal, child, and reproductive-health questions a nurse-midwife asks, and, to our knowledge, no public chunk-level relevance benchmark exists for maternal-health guideline retrieval. We release two benchmarks that fill these gaps. mamabench is a scope-filtered QA set of 25,949 items assembled from seven existing expert-authored sources across multiple-choice, short-answer, and rubric-graded tracks; to help users calibrate the LLM judge that scores the rubric track, we re-scope HealthBench's physician-labelled meta-evaluation to the domain. mamaretrieval pairs 3,185 clinical queries with graded (0-6) relevance labels over a 63,650-chunk maternal-health guideline corpus, using a decomposed rubric that distinguishes a chunk that answers a query from one merely on its topic. Three decisions shape both: assemble and filter expert sources rather than author questions, grade relevance rather than binarise it, and measure and disclose the limits of the labels -- scope-classifier agreement, a frontier-judge check, and a pooling-completeness audit -- rather than treat them as an oracle. A companion paper uses the benchmarks to evaluate a deployed on-device assistant; both are released openly for research.
