Search papers, labs, and topics across Lattice.
This paper critiques the use of surface-level similarity metrics like BLEU in evaluating retrieval-augmented personalized dialogue systems, arguing they fail to capture coherence and consistency. They re-evaluate the LAPDOG framework using human and LLM-based judges, revealing issues like corrupted dialogue histories and contradictory information. The study demonstrates a strong alignment between human and LLM judgments, highlighting the necessity for cognitively grounded evaluation methods in dialogue systems.
Surface-level metrics like BLEU are misleading for evaluating dialogue systems, as human and LLM judges reveal critical flaws in coherence and consistency that these metrics miss entirely.
In cognitive science and linguistic theory, dialogue is not seen as a chain of independent utterances but rather as a joint activity sustained by coherence, consistency, and shared understanding. However, many systems for open-domain and personalized dialogue use surface-level similarity metrics (e.g., BLEU, ROUGE, F1) as one of their main reporting measures, which fail to capture these deeper aspects of conversational quality. We re-examine a notable retrieval-augmented framework for personalized dialogue, LAPDOG, as a case study for evaluation methodology. Using both human and LLM-based judges, we identify limitations in current evaluation practices, including corrupted dialogue histories, contradictions between retrieved stories and persona, and incoherent response generation. Our results show that human and LLM judgments align closely but diverge from lexical similarity metrics, underscoring the need for cognitively grounded evaluation methods. Broadly, this work charts a path toward more reliable assessment frameworks for retrieval-augmented dialogue systems that better reflect the principles of natural human communication.
