Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
This work evaluates frontier model distillation defenses under an updated threat model where adversaries apply post-distillation reinforcement learning to student models rather than relying solely on supervised trace imitation. The authors show that downstream RL readily repairs performance degradation from perturbed or incomplete traces obtainable via standard APIs, effectively nullifying defenses that appear robust immediately post-distillation. Consequently, any defense that allows adversaries to reconstruct even approximate reasoning traces fails to protect closed-source capabilities against determined adversaries.
Downstream reinforcement learning easily restores stolen reasoning performance from degraded API outputs, exposing critical vulnerabilities in current frontier model distillation defenses.
Distillation attacks copy the reasoning capabilities of closed-source large language models, allowing bad actors to replicate state-of-the-art performance at low cost. Attackers systematically collect a large volume of frontier model reasoning traces and then train (i.e., "distill") their own models on these traces. Existing defenses against distillation attacks are typically evaluated immediately after distillation, implicitly assuming attackers do not train their models any further. In this paper, we argue that a more realistic threat model includes further training with reinforcement learning after distillation. A misspecified threat model can give a false sense of security -- some defenses that seem effective after distillation can be broken after subsequent reinforcement learning. Practically, reinforcement learning lowers the bar for a distillation attack to be effective. We show that simple attacks can steal reasoning capabilities from existing closed-source language models using data easily obtainable from current APIs, yielding reasoning improvements equivalent to more sophisticated attacks that extract the full hidden traces. Results indicate that any distillation defense that leaks sufficient information to reconstruct approximate reasoning traces is likely ineffective. We conclude by discussing broader implications and batch-level distillation defenses which could be more effective.