Search papers, labs, and topics across Lattice.
This paper introduces a case-aware LLM-as-a-Judge evaluation framework tailored for multi-turn, enterprise RAG systems, addressing the limitations of existing benchmarks in capturing enterprise-specific failure modes. The framework uses eight operationally grounded metrics to evaluate retrieval quality, grounding, answer utility, precision, and case/workflow alignment at each turn. Results demonstrate that the proposed framework provides more actionable insights for system improvement compared to generic proxy metrics by exposing enterprise-critical tradeoffs in short and long workflows.
Current RAG evaluation metrics miss critical enterprise failure modes, but this new framework exposes actionable tradeoffs in multi-turn workflows by evaluating retrieval, grounding, utility, precision, and case alignment.
Enterprise Retrieval-Augmented Generation (RAG) assistants operate in multi-turn, case-based workflows such as technical support and IT operations, where evaluation must reflect operational constraints, structured identifiers (e.g., error codes, versions), and resolution workflows. Existing RAG evaluation frameworks are primarily designed for benchmark-style or single-turn settings and often fail to capture enterprise-specific failure modes such as case misidentification, workflow misalignment, and partial resolution across turns. We present a case-aware LLM-as-a-Judge evaluation framework for enterprise multi-turn RAG systems. The framework evaluates each turn using eight operationally grounded metrics that separate retrieval quality, grounding fidelity, answer utility, precision integrity, and case/workflow alignment. A severity-aware scoring protocol reduces score inflation and improves diagnostic clarity across heterogeneous enterprise cases. The system uses deterministic prompting with strict JSON outputs, enabling scalable batch evaluation, regression testing, and production monitoring. Through a comparative study of two instruction-tuned models across short and long workflows, we show that generic proxy metrics provide ambiguous signals, while the proposed framework exposes enterprise-critical tradeoffs that are actionable for system improvement.
