Search papers, labs, and topics across Lattice.
This paper introduces CopyShield, a benchmark designed to evaluate the effectiveness of three copyright defenses in large language models (LLMs) at different intervention levels: contrastive decoding, Direct Preference Optimization (DPO), and activation intervention. The study reveals that while contrastive decoding maintains low degeneracy, it struggles with literal suppression, whereas DPO significantly reduces literal leakage but suffers from high paraphrase-loop degeneracy. Activation intervention emerges as the most effective method for minimizing non-literal leakage while maintaining utility, highlighting the complex trade-offs between compliance and performance in copyright defense strategies for LLMs.
Targeted non-literal suppression in LLMs reveals a critical trade-off between compliance and output quality that could redefine copyright defense strategies.
Large language models can reproduce memorized text verbatim, yet copyright defenses are usually evaluated under incompatible protocols. We introduce CopyShield, a controlled benchmark comparing three representative defenses at distinct intervention levels: contrastive decoding (output), Direct Preference Optimization (behavioral), and activation intervention (representation). We evaluate CopyShield on two model families, LLaMA-3.1-8B and Mistral-7B-v0.3, using controlled memorization over five public-domain books and a shared protocol measuring literal leakage, calibrated non-literal leakage, utility, and degeneracy. Across these methods, intervention level is associated with distinct compliance-utility trade-offs. On LLaMA-3.1-8B, contrastive decoding remains near-degeneracy-free (0-2%) but reaches a literal-suppression floor at NV-Recall 0.192-0.203. DPO nearly eliminates literal leakage (0.263 to 0.002) but induces paraphrase-loop degeneracy in 58% of QA outputs, with no utility gain over the SFT baseline. Activation intervention attains the lowest non-literal flagging rate (1/200) by blocking 84% of non-literal queries before generation. Human evaluation confirms that DPO has low coherence, whereas activation lowers perceived copyright risk through broad refusal. On Mistral-7B-v0.3, the output- and representation-level patterns persist, while DPO degeneracy falls to 10-14%, showing that its severity is model-dependent. Together, CopyShield provides cross-level reference baselines and identifies targeted non-literal suppression as an open challenge. The code is available at https://github.com/spotai-mbzuai/CopyShield.git.
