Search papers, labs, and topics across Lattice.
This paper investigates geometric hallucination types in language models by applying PCA-whitening and eigenspectrum decomposition to the embedding space of GPT-2-small. Whitening reveals that cluster commitment, measured by maximum cluster alignment (max\_sim), effectively separates Type 2 (wrong-well convergence) from Type 3 (coverage gaps) hallucinations. The analysis also suggests that distinguishing Type 1 (center-drift) from Type 2 hallucinations is a capacity limitation, not a measurement artifact, and highlights the sensitivity of micro-signals to prompt set variations.
Whitening the embedding space of GPT-2-small exposes cluster commitment as the key geometric property separating different types of language model hallucinations.
A geometric hallucination taxonomy distinguishes three failure types -- center-drift (Type~1), wrong-well convergence (Type~2), and coverage gaps (Type~3) -- by their signatures in embedding cluster space. Prior work found Types~1 and~2 indistinguishable in full-dimensional contextual measurement. We address this through PCA-whitening and eigenspectrum decomposition on GPT-2-small, using multi-run stability analysis (20 seeds) with prompt-level aggregation. Whitening transforms the micro-signal regime into a space where peak cluster alignment (max\_sim) separates Type~2 from Type~3 at Holm-corrected significance, with condition means following the taxonomy's predicted ordering: Type~2 (highest commitment) $>$ Type~1 (intermediate) $>$ Type~3 (lowest). A first directionally stable but underpowered hint of Type~1/2 separation emerges via the same metric, generating a capacity prediction for larger models. Prompt diversification from 15 to 30 prompts per group eliminates a false positive in whitened entropy that appeared robust at the smaller set, demonstrating prompt-set sensitivity in the micro-signal regime. Eigenspectrum decomposition localizes this artifact to the dominant principal components and confirms that Type~1/2 separation does not emerge in any spectral band, rejecting the spectral mixing hypothesis. The contribution is threefold: whitening as preprocessing that reveals cluster commitment as the theoretically correct separating metric, evidence that the Type~1/2 boundary is a capacity limitation rather than a measurement artifact, and a methodological finding about prompt-set fragility in near-saturated representation spaces.
