Search papers, labs, and topics across Lattice.
This study evaluates extraction methods as a safer alternative to rewriting in retrieval-augmented generation (RAG) systems for accessing organizational documentation, particularly in safety-critical contexts. By comparing various prompting strategies, including line-number-based source selection and multi-stage pipelines, the authors find that line-number selection significantly outperforms other methods in terms of recall and alignment with source text. The results indicate that while safety-oriented approaches enhance precision, they also lead to systematic omissions, highlighting the importance of method selection based on document structure.
Line-number extraction outperforms rewriting strategies, achieving up to 95% term recall while minimizing hallucinations in safety-critical applications.
Large language models (LLMs) are increasingly used to access organisational documentation, including standard operating procedures (SOPs), HR policies and institutional guidelines. However, retrieval-augmented generation (RAG) systems that rely on free-form rewriting can introduce hallucinations and unstable trade-offs between completeness and conciseness, particularly in safety- and compliance-critical settings. Objectives: To evaluate extraction as a hallucination-resistant alternative to rewriting-based RAG and compare strategies that balance precision, recall and safety across document types and model scales. Methods: We compare multiple prompting strategies, including line-number-based source selection, extraction of relevant guideline sentences with explicit safety annotations, and a multi-stage pipeline that refines draft answers using supporting evidence from source guidelines. Experiments are conducted on documents of varying length and structure, including local NHS acute care and oncology guidelines and UK-wide NICE guidelines, using both frontier-scale and locally deployable models. Performance is assessed using automatic metrics and human expert evaluation of relevance and completeness. Results: Line-number selection achieves the strongest results, outperforming direct copying and safety-focused strategies across both large and small models while maintaining high term recall (up to 95%) and close alignment with source text. Safety-oriented approaches improve precision but introduce systematic omissions, while multi-stage filtering further amplifies this trade-off. Performance varies with document structure: line-based extraction excels in protocol-like content, whereas alternative strategies perform better on more verbose documents (up to 97% term recall).
