Search papers, labs, and topics across Lattice.
This paper audits three state-of-the-art content moderation systems (OpenAI, Meta, Google) to assess their impact on LLMs used in therapy. The study finds that these systems frequently flag content from real therapy sessions as undesirable, potentially hindering the LLM's ability to address sensitive topics. This highlights a critical tension between safety guardrails and the therapeutic utility of LLMs.
Content moderation systems, designed to protect users, may inadvertently cripple LLMs' ability to function as effective therapists by censoring crucial topics in real therapy sessions.
Large language models (LLMs) are increasingly being used for emotional support. They are also being developed for formal therapy purposes. However, LLMs like ChaptGPT or Llama are often developed with content moderation guardrails that prevent them from discussing sensitive subjects with users for both liability and safety purposes, and this inability to broach these subjects may affect their capacity as therapists. In this study, we perform an algorithm audit on three state-of-the-art moderation systems (OpenAI's moderation endpoint, Meta's Llama Guard, and Google's Shield Gemma) to investigate the extent to which these systems flag the content of real-life therapy sessions as undesirable. Our results raise implications for the limitations that users and organizations may encounter when designing LLMs to play the part of a therapist.
