Search papers, labs, and topics across Lattice.
This paper investigates the issue of large language models (LLMs) generating unsafe responses in task-oriented dialogue when backend database calls fail or return incorrect information. By employing a lightweight prompting-based recovery approach, the authors demonstrate that their Guided-Retry strategy significantly reduces hallucinations in LLM responses without necessitating retraining or additional model calls. The results show a marked improvement, with hallucination rates dropping by up to 50% across multiple model families and diverse database conditions, although some residual hallucination remains a challenge.
Hallucinations in task-oriented dialogue can be cut by nearly half using a simple guided recovery prompt, even in the face of database failures.
Large language models used in task-oriented dialogue often produce fluent but unsafe responses when backend database calls fail, return empty results, or surface mismatched information, inventing venues, confirmations, or booking details not grounded in the database. We study a lightweight prompting-based recovery approach that improves robustness without retraining or additional model calls. We compare three response strategies, including a guided recovery prompt conditioned on structured database status, across six open-weight model families (DeepSeek-R1, Gemma-2, Llama-3, Mistral, Phi-3, and Qwen-2.5) and four database conditions: empty result, wrong-domain retrieval, API error, and clean retrieval. Using fault-injected benchmarks built on two structurally different datasets, MultiWOZ 2.2 (5 domains) and SGD (20 domains), we find that naive agents hallucinate on 30.5% of failure turns on MultiWOZ and 20.9% on SGD. Our Guided-Retry strategy reduces hallucination by 50% on MultiWOZ (30.5 to 15.3%) and by 42% on SGD (20.9 to 12.2%) without retraining. However, residual hallucination remains substantial (6-37% across models), with wrong-domain failures the hardest case. Results are consistent across both datasets and all six model families, and human annotation shows substantial agreement while supporting the validity of the automatic commitment-safety metric.
