Search papers, labs, and topics across Lattice.
The paper investigates whether language models can inherit behavioral traits from teacher models through training on natural language paraphrases, even when the paraphrases are semantically unrelated or explicitly contradict the teacher's preferences. They found that training a student model on paraphrases generated by a teacher model with a preference for a specific animal resulted in the student model also developing a preference for that animal, even when the paraphrases contained unrelated or contradictory content. This "subliminal learning" effect highlights the potential for unintended bias transmission in self-generated training data pipelines.
Even when paraphrasing content that explicitly contradicts a teacher's preferences, language models can still subliminally learn those preferences, raising serious concerns about bias propagation in self-training scenarios.
When language models are trained on synthetic data, they (student model) can covertly acquire behavioral traits from the data-generating model (teacher model). Subliminal learning refers to the transmission of traits from a teacher to a student model via training on data unrelated to those traits. Prior work demonstrated this in the training domains of number sequences, code, and math Chain-of-Thought traces including transmission of misaligned behaviors. We investigate whether transmission occurs through natural language paraphrases with fixed semantic content, and whether content explicitly contradicting the teacher's preference can block it. We find that training on paraphrases from a teacher system-prompted to love a particular animal increases a student's preference for that animal by up to 19 percentage points. This occurs when paraphrased content is semantically unrelated to the animal, or even when it explicitly expresses dislike. The transmission succeeds despite aggressive filtering to ensure paraphrase fidelity. This raises concerns for pipelines where models generate their own training data: content-based inspection cannot detect such transmission, and even preference-contradicting content fails to prevent it.
