Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper reviews empirical evidence regarding user interactions with organization-backed AI advisors, focusing on the critical constructs of skepticism, verification, and reliance in the context of AI-generated information. The findings reveal that while reliance on AI outputs is commonly measured, skepticism and verification are often conflated and inadequately assessed, leading to mixed results regarding the effectiveness of interventions aimed at prompting user scrutiny. Notably, interventions that warn users about hallucination risks show limited impact, highlighting a gap in understanding how users engage with potentially inaccurate AI responses across various domains such as healthcare and content generation.
Users often fail to recognize the need for verification in AI-generated advice, even as reliance on these outputs remains high.
Generative AI systems are increasingly used by organizations to deliver information to consumers, patients, students, employees, and citizens. These systems can hallucinate, producing plausible but inaccurate responses. A central question for AI-advised decisions is therefore not only whether users rely on inaccurate information, but whether they recognize that a response may require verification. To answer this question, we review emerging empirical evidence relevant to hallucination detection in goal-directed interactions, with a focus on organization-backed AI advisors. We distinguish three constructs that existing studies often conflate: whether users are skeptical of information presented, whether they check it, whether checking succeeds, and whether the result of user verification affects reliance on the information. Across studies examining product search, medical decision-making, content generation, and chatbot-assisted tasks, several patterns emerge. Nearly all studies measure reliance, while variables such as user skepticism and verification of the information are more often targeted by an intervention than measured directly. The cues used to prompt scrutiny of the AI response are predominantly related to the AI output, such as source citations, and the most deployable of these AI output interventions for organizations (general and specific warnings about the risk of hallucinations) show the weakest and most mixed effects in the studies reviewed. Although the existing literature posits that users may be more likely to scrutinize responses related to particular areas of content, no studies varied the content category, leaving this question open for further research. In future research, measuring skepticism and verification separately from reliance may clarify what current evidence shows, what it only implies, and which questions require further exploration.