Search papers, labs, and topics across Lattice.
This paper introduces an Evidence-Grounded Customer-Service Agent Workflow that enhances the performance of customer-service bots by integrating various evidence retrieval methods and policy-guided orchestration. The approach employs a combination of BM25 recall, vector recalls, and cross-encoder reranking to ensure that LLM decisions remain within defined evidence boundaries and policy rules. Key findings include the development of three reusable deployment patterns that improve the quality of responses while maintaining compliance with operational safeguards, ultimately leading to more effective customer interactions.
Grounded in robust evidence retrieval, this workflow ensures customer-service LLMs adhere to policy rules while significantly enhancing answer quality.
Production customer-service bots must improve answer quality across iterative releases, yet large language models must not bypass evidence boundaries, policy rules, or human-handoff safeguards. We present an \textbf{Evidence-Grounded Customer-Service Agent Workflow} deployed in a real-world customer-service setting. BM25 recall, issue-title-vector recall, issue-description-vector recall, weighted RRF fusion, and cross-encoder reranking construct grounded FAQ evidence for controlled LLM decisions. Policy-guided orchestration then combines this RAG evidence with scenario-specific rule evidence, conversation memory, and clarification state inside a fixed LangGraph DAG~\cite{langgraph2024}. The paper contributes three reusable deployment patterns: \textbf{hybrid RAG evidence construction}, where multi-channel retrieval and reranking produce auditable FAQ candidates; \textbf{evidence-grounded issue/action decision}, where an Evidence-Grounded Decision Module selects an issue/action from typed FAQ evidence and scenario-specific rule evidence; and \textbf{trace-driven RAG and reranker improvement}, where traces diagnose whether failures come from recall, ranking, final candidate selection, clarification, rule-derived evidence, or action policy, and where reranker fine-tuning is evaluated not only for in-domain gain but also for forgetting risk.
