Search papers, labs, and topics across Lattice.
This paper introduces the concept of "linguistic illegibility," highlighting the disconnect between the linguistic outputs of large language models (LLMs) and their internal computational processes. It argues that reliance on linguistic self-reporting for security mechanisms is fundamentally flawed, as these outputs may not accurately reflect the model's internal workings. The authors propose taint tracking as a promising alternative for ensuring model safety, suggesting that it can effectively isolate system states from model-generated data regardless of linguistic interpretation.
Security mechanisms based on LLM linguistic outputs are fundamentally unreliable due to inherent "linguistic illegibility," necessitating new isolation techniques for model safety.
LLMs are trained to generate natural language. However, various strands of evidence indicate that an LLM's externalized linguistic outputs and mechanistically-extracted linguistic features can be an unreliable lens for understanding internal model computation. We introduce the term ``linguistic illegibility''to broadly refer to scenarios in which an LLM's externalized or mechanistically-probed language artifacts fail to represent how the model actually thinks. We argue that the specter of linguistic illegibility is unavoidable for LLMs whose internal computations are not directly expressed via language, but rather math over activation spaces (with lossy translations between activation spaces and natural language happening at the bookends). If linguistic illegibility is always possible, then security mechanisms that rely on a model's linguistic self-reporting (e.g., chain-of-thought monitoring, constitutional self-critique, activation probing for linguistically-defined feature vectors) can never be completely sound; the model sandbox will always need isolation techniques whose guarantees do not depend on reading a model's linguistic state at all. We argue that observing a model's outputs using taint tracking is a promising approach for an effective sandbox: regardless of how a model linguistically self-reports, a taint tracking policy can define, a priori, various pieces of system state that should never be influenced by model-produced data. We also discuss several additional sandboxing mechanisms (e.g., robust virtualization, third-party auditing of sandboxing configurations) which collectively provide a critical floor beneath linguistic monitoring, and would have mitigated recent sandbox exploits by frontier models.
