Search papers, labs, and topics across Lattice.
The paper introduces DistillGuard, a framework for systematically evaluating output-level defenses against LLM knowledge distillation attacks. It categorizes defenses into output perturbation, data poisoning, and information throttling, then evaluates nine defense configurations using Qwen3-14B as the teacher and Qwen2.5-7B-Instruct as the student across MATH-500, HumanEval+, and MT-Bench. The results show that most output-level defenses are ineffective against a naive attacker, with only chain-of-thought removal substantially impairing mathematical reasoning.
Most output-level defenses against LLM knowledge distillation are surprisingly weak, failing to prevent knowledge theft even from naive attackers.
Knowledge distillation from proprietary LLM APIs poses a growing threat to model providers, yet defenses against this attack remain fragmented and unevaluated. We present DistillGuard, a framework for systematically evaluating output-level defenses against LLM knowledge distillation. We introduce a taxonomy of three defense categories -- output perturbation, data poisoning, and information throttling -- and evaluate nine defense configurations using a standardized pipeline with Qwen3-14B as teacher and Qwen2.5-7B-Instruct as student across three benchmarks (MATH-500, HumanEval+, MT-Bench). Our results reveal that, in a same-family distillation setting against a naive attacker, most output-level defenses are surprisingly ineffective: paraphrasing-based perturbation barely degrades distilled student quality, and data poisoning primarily impairs conversational fluency while leaving task-specific capabilities intact. Only chain-of-thought removal substantially impairs mathematical reasoning (31.4\% vs.\ 67.8\% baseline), though code generation remains unaffected. These findings demonstrate that the effectiveness of distillation defenses is highly task-dependent and that current output-level approaches are insufficient to broadly prevent knowledge theft.
