Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The paper investigates why language models exhibit a "truth bias" even when trained on noisy data, proposing the Compression-Consistency Principle: models favor hypotheses enabling shorter, more consistent data descriptions. Using GPT-2-style transformers on synthetic math corpora, the study demonstrates that models prefer correct completions when errors are random (83.1% accuracy on balanced data), but this preference diminishes with coherent, incorrect rule systems. Embedding verification steps restores the preference for correctness, suggesting that compression and consistency, rather than an intrinsic drive for truth, drive the observed bias.
Language models seem to prefer truth not because they're seeking it, but because correct information is often easier to compress and more internally consistent.
Why do language models sometimes prefer correct statements even when trained on mixed-quality data? We introduce the Compression--Consistency Principle: next-token prediction favors hypotheses that allow shorter and more internally consistent descriptions of the training data. Truth bias emerges only when false alternatives are structurally harder to compress. We test this using small GPT-2-style character-level transformers (3.5M--86M parameters) on synthetic math corpora with controlled mixtures of correct and incorrect rules. In the random-error setting, models strongly prefer correct completions in paired evaluation: 83.1% accuracy at balanced data and 67.0% even when correct rules appear in only 10% of the corpus. Replacing random errors with a coherent but mathematically incorrect rule system largely eliminates the preference (near-chance accuracy). In a more natural-language-like synthetic world, the effect is weaker but still present (57.7%). Additional experiments show that embedding verification steps can restore preference for correctness even at small scale, while increasing the number of consistent rules produces a graded improvement in accuracy. Our results suggest that what appears as a "truth bias" is largely a side effect of compression pressure and preference for internal consistency, rather than an intrinsic drive toward truth. Full code and data are available at https://github.com/Rai220/compression-drives-truth.