Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates distillation inference, a method to detect if a suspect LLM was trained by distilling a proprietary teacher model. The approach involves training shadow models—one distilled from the teacher's reasoning traces and another trained on reference answers—to establish expected behaviors under each hypothesis. By comparing the suspect model's output alignment with the teacher's reasoning traces against these shadow models, the auditor can generate a calibrated p-value, achieving perfect detection in a preliminary study.
Distillation inference can reliably detect unauthorized LLM distillation by comparing suspect model outputs against teacher reasoning traces via shadow models.
Unauthorized model distillation, in which a model is trained on the outputs of a proprietary large language model (LLM), is a growing threat to model providers. We study distillation inference: determining whether a suspect model was distilled from another model or trained independently. We formulate this problem as a hypothesis test and estimate the behavior expected under each hypothesis by training shadow models: distilled shadow models learn from the teacher's reasoning traces, whereas independent shadow models learn only from reference answers. The auditor measures how closely each model predicts the teacher's reasoning outputs and then uses the shadow models to convert the suspect's score into a calibrated p-value. In a preliminary study using Qwen2.5-7B as the teacher and Llama-3.2-3B for the suspects, our test achieves a true positive rate of 1.0 at a significance level of 0.02. These results demonstrate the feasibility of using distillation inference to detect distillation attacks.