Search papers, labs, and topics across Lattice.
This paper introduces a psychometric framework for auditing latent biases in LLMs, moving beyond transient task accuracy to identify durable, provider-level behavioral signatures. The framework uses forced-choice ordinal vignettes with semantically orthogonal decoys and cryptographic permutation-invariance to quantify tendencies like optimization bias and sycophancy without ground-truth labels. Results from auditing nine leading models reveal a significant "lab signal," indicating that persistent biases are embedded during training and alignment, potentially leading to compounding risks in multi-layered AI architectures.
LLMs exhibit persistent "lab signals" in their latent biases, suggesting that alignment choices made by AI providers can create compounding ideological echo chambers in multi-layered AI systems.
As Large Language Models (LLMs) transition from standalone chat interfaces to foundational reasoning layers in multi-agent systems and recursive evaluation loops (LLM-as-a-judge), the detection of durable, provider-level behavioral signatures becomes a critical requirement for safety and governance. Traditional benchmarks measure transient task accuracy but fail to capture stable, latent response policies -- the ``prevailing mindsets'' embedded during training and alignment that outlive individual model versions. This paper introduces a novel auditing framework that utilizes psychometric measurement theory -- specifically latent trait estimation under ordinal uncertainty -- to quantify these tendencies without relying on ground-truth labels. Utilizing forced-choice ordinal vignettes masked by semantically orthogonal decoys and governed by cryptographic permutation-invariance, the research audits nine leading models across dimensions including Optimization Bias, Sycophancy, and Status-Quo Legitimization. Using Mixed Linear Models (MixedLM) and Intraclass Correlation Coefficient (ICC) analysis, the research identifies that while item-level framing drives high variance, a persistent ``lab signal'' accounts for significant behavioral clustering. These findings demonstrate that in ``locked-in'' provider ecosystems, latent biases are not merely static errors but compounding variables that risk creating recursive ideological echo chambers in multi-layered AI architectures.
