Search papers, labs, and topics across Lattice.
This paper introduces LayerRAG-Bench, a comprehensive benchmark designed to evaluate the reliability of agentic retrieval-augmented generation systems across multiple layers of functionality. The study reveals that while schema normalization significantly improves success rates in addressing schema drift, it fails to rectify issues related to stale evidence, missing tool outputs, and session context errors. The findings advocate for a layer-specific evaluation approach, emphasizing that reliability interventions must be accurately credited to their respective layers rather than assumed to provide universal solutions.
Schema normalization boosts schema-drift success to 91.3%, yet fails to address critical issues like stale evidence and session context errors in retrieval-augmented generation systems.
Agentic retrieval-augmented generation systems can produce answers that appear grounded while failing at the evidence, tool-contract, authorization, or session-state layer. We introduce LayerRAG-Bench, a controlled cross-layer reliability benchmark with 8 enterprise domains, 240 tasks, 9 fault scenarios, 2 contract modes, and 38,880 live task-level records across nine models from OpenAI, Anthropic, and Gemini. Schema normalization raises schema-drift success from 0.000 to 0.913, but stale evidence, missing tool output, denied permissions, and wrong-session context are not recovered by schema normalization. Groundedness-only evaluation also produces substantial false positives under stale and wrong-session evidence. These results support a layer-specific evaluation principle: a reliability intervention should be credited for repairing its target layer without being mistaken for a universal fix.
