Search papers, labs, and topics across Lattice.
This study enhances public health question answering by applying Retrieval-Augmented Generation (RAG) to large language models (LLMs), addressing issues of hallucination and the need for up-to-date information. By extending the PubHealthBench benchmark and systematically evaluating various retrieval methods, the authors demonstrate that hybrid retrieval significantly improves recall and ranking quality, allowing smaller models to outperform larger ones when retrieval is effectively utilized. The introduction of a rubric-based evaluation for free-form answers reveals critical insights into the dimensions of faithfulness and completeness, underscoring the importance of retrieval quality in public health applications.
Hybrid retrieval methods can enable smaller models to outperform larger ones in public health question answering, fundamentally shifting the landscape of LLM utility in this domain.
Large language models (LLMs) achieve promising results on medical question answering benchmarks, yet their use in public health is constrained by hallucinations and the rapid evolution of official guidance. Retrieval-Augmented Generation (RAG) mitigates these risks by grounding responses in an explicitly maintained corpus, but end-to-end performance depends critically on retrieval configuration and on evaluation beyond multiple-choice formats. We extend PubHealthBench, a question answering (QA) benchmark of 7,929 questions derived from UK Government public health guidance, into a retrieval-augmented setting and systematically evaluate retrieval and generation choices. We compare dense, sparse, and hybrid retrieval across multiple embedding models and corpus variants, and show that hybrid retrieval consistently improves recall and ranking quality, with chunk length and topic interacting with ranking performance. Providing retrieved context substantially increases multiple-choice accuracy across a diverse set of LLMs, enabling smaller open-weight models to match or outperform larger models used without retrieval, with gains primarily driven by retrieval quality and careful context selection. To assess realistic free-form answering, we introduce a rubric-based LLM-as-a-judge covering faithfulness, completeness, clarity, and factual consistency, and validate it against dual human annotations. Judge-human agreement is strongest for faithfulness and completeness, while factual consistency and clarity are less reliably reproduced, motivating caution when interpreting those dimensions at scale. Overall, our results highlight retrieval as a primary lever for reliable public health QA and provide practical guidance for building and evaluating RAG systems grounded in official guidance.
