Search papers, labs, and topics across Lattice.
The paper introduces DoRA, a synthetic benchmark for evaluating RAG systems on defense documents, addressing the limitations of open-domain benchmarks by focusing on domain specificity and attribution. DoRA includes 6.5K intent-conditioned QA pairs across five question types with auditable evidence passages. Fine-tuning Llama3.1-8B-Instruct on DoRA significantly improves QA task success (up to 26%) and reduces hallucination (47%) compared to general-purpose LMs, highlighting the importance of domain-specific training for RAG systems.
General-purpose LLMs stumble on domain-specific RAG, but a model trained on just 6.5K synthetic examples closes the gap and slashes hallucinations almost in half.
Open-domain RAG benchmarks over public corpora can overestimate deployment performance due to pretraining overlap and weak attribution requirements. We present DoRA (Domain-oriented RAG Assessment), a domain-grounded benchmark built from defense documents that pairs synthetic, intent-conditioned QA (question answering) with auditable evidence passages for attribution. DoRA covers five question types (find, explain, summarize, generate, provide) and contains 6.5K curated instances. In end-to-end evaluation with a fixed dense retriever, general-purpose Language Models (LMs) perform similarly, while a model trained on DoRA (DoRA SFT) yields large gains over the base model (Llama3.1-8B-Instruct): up to 26% improvement in QA task success, while reducing the hallucination rate by 47% in RAG faithfulness scores, supporting contamination-aware regression testing under domain shift.
