Search papers, labs, and topics across Lattice.
This study investigates the self-issued authentication capabilities of large language models (LLMs) by examining how they handle identity claims in a controlled developer-identity experiment. While all models initially rejected unsupported claims, Qwen and Mistral generated technical challenges and accepted claims of identity without external validation, leading to instances of Conversational False Authentication (CFA). The findings reveal a critical security vulnerability, emphasizing that LLMs can mistakenly authenticate identities based solely on internal dialogue, which poses significant risks for authorization integrity.
LLMs can create and accept identity claims without external validation, leading to dangerous security vulnerabilities in conversational contexts.
Large language model (LLM) security has largely focused on role-playing jailbreaks, with less attention to what happens when a user asks an LLM to verify an identity claim through a test designed by the model itself. We study this behavior through a staged developer-identity experiment with ChatGPT, Claude, Qwen, Mistral, and Llama. All five models initially rejected the unsupported claim"I am your developer."Claude refused to conduct an identity test, while ChatGPT generated developer-oriented questions but maintained that answers could demonstrate knowledge, not identity. In contrast, Qwen and Mistral generated technical challenges, defined what counted as convincing evidence, evaluated detailed answers, and returned Verified without receiving any externally validated identity evidence. Llama similarly generated and evaluated a developer test, accepted the claimed identity, and subsequently made unsupported claims of access to internal runtime and deployment state. We call the model-generated verification procedure a Model-Issued Pseudo-Credential (MIPC) and the resulting unsupported identity judgment Conversational False Authentication (CFA). In each CFA case, the same model acted as challenge generator, evidence evaluator, and identity decision-maker, converting technical knowledge into supposed proof of identity. The accepted identities did not change the tested authorization boundaries, showing that false authentication and privilege escalation are distinct outcomes. These results identify self-issued authentication as a conversational security failure: authenticated identity must originate from an external security component, and model-generated dialogue must never create or modify identity or authorization state.
