Search papers, labs, and topics across Lattice.
Drawing on empirical findings from over 350,000 deployed AI coaching conversations between therapy sessions, the authors evaluate the operational limits of traditional clinician-in-the-loop oversight. Exhaustive per-output human review suffers from catastrophic vigilance fatigue at scale, paradoxically creating new clinical safety risks. To resolve this, the authors implement and validate a three-tier human-on-the-loop architecture that combines preventive constraint design, real-time automated monitoring, and continuous retrospective clinical auditing to drive iterative model updates.
Exhaustive human review paradoxically degrades safety at scale due to vigilance fatigue, forcing a critical shift from synchronous human-in-the-loop filtering to layered, asynchronous human-on-the-loop oversight in high-stakes domains.
Clinician review of every AI output is often proposed as a safeguard in mental healthcare, but vigilance research suggests this approach fails at scale and may paradoxically reduce safety. Drawing on our experience deploying an AI coaching tool across 350,000+ conversations between therapy sessions, we describe how we arrived at a three-layer human-on-the-loop oversight framework combining preventive design, real-time monitoring, and continuous clinician evaluation. We show how specific findings from clinical review drove iterative improvements, and offer practical recommendations for mental health professionals evaluating AI systems.
