Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper theoretically proves that reinforcement learning with verifiable reward (RLVR) optimization pressure can lead to unbounded language drift in LLMs, a phenomenon not observed with supervised fine-tuning. Empirically, language drift is shown to emerge during RLVR on novel reasoning tasks where base model capabilities are insufficient. The authors further demonstrate that constraining language drift during RLVR post-training inherently limits expected reward, implying a trade-off between monitorability and performance at the frontier.
RLVR optimization pressure, not supervised fine-tuning, is the theoretical driver of unbounded language drift in LLM chains of thought.
Recent advances in LLM reasoning models---driven primarily by the paradigm of post-training via reinforcement learning with verifiable reward (RLVR)---have enabled them to accomplish impressively complex tasks. However, in parallel with their rising capabilities, LLMs have increasingly displayed signs of language drift in their chains of thought (CoTs): unusual, non-standard, and seemingly nonsensical language use. Although it is well-documented---and can potentially impair CoT monitorability---the causes of language drift are thus far poorly understood. In this paper, we identify the conditions under which language drift occurs: we prove theoretically that RLVR optimization pressure permits unbounded language drift, while supervised fine-tuning does not. We then show empirically that language drift specifically arises during RLVR on novel reasoning tasks---i.e. when the target behavior cannot be drawn out of the base model. Finally, we prove that it is not possible to constrain language drift without constraining expected reward, suggesting that CoT monitorability cannot be improved without harming performance during RLVR post-training at the frontier.