Search papers, labs, and topics across Lattice.
This paper evaluates the effectiveness of LLMs in delivering Cognitive Behavioral Therapy (CBT) by comparing generation-only and Retrieval-Augmented Generation (RAG) approaches against anonymized transcripts of real therapy sessions. The study assesses linguistic quality, semantic coherence, and therapeutic fidelity using NLG metrics, NLI, and automated skills assessment. Results show that while LLMs can generate CBT-like dialogues, they struggle with empathy and consistency compared to human therapists.
LLMs can mimic CBT dialogues, but still fail to convey the empathy and consistency of a human therapist.
As mental health issues continue to rise globally, there is an increasing demand for accessible and scalable therapeutic solutions. Many individuals currently seek support from Large Language Models (LLMs), even though these models have not been validated for use in counseling services. In this paper, we evaluate LLMs' ability to emulate professional therapists practicing Cognitive Behavioral Therapy (CBT). Using anonymized, transcribed role-play sessions between licensed therapists and clients, we compare two approaches: (1) a generation-only method and (2) a Retrieval-Augmented Generation (RAG) approach using CBT guidelines. We evaluate both proprietary and open-source models for linguistic quality, semantic coherence, and therapeutic fidelity using standard natural language generation (NLG) metrics, natural language inference (NLI), and automated scoring for skills assessment. Our results indicate that while LLMs can generate CBT-like dialogues, they are limited in their ability to convey empathy and maintain consistency.
