Search papers, labs, and topics across Lattice.
The paper introduces Private Mask Pre-Training (PMP), a pre-training framework designed to create foundation models that are broadly usable but resistant to unauthorized fine-tuning. PMP concentrates representation learning into a sparse, privately masked subnetwork, releasing only the final dense weights. This induces a mismatch between the fine-tuning objective and the pre-training geometry for those without the mask, thereby limiting adaptation gains.
Foundation models can be made intrinsically resistant to unauthorized fine-tuning by concentrating learning in a sparsely masked subnetwork that remains private.
Open-sourcing foundation models (FMs) enables broad reuse but also exposes model trainers to economic and safety risks from unrestricted downstream fine-tuning. We address this problem by building non-fine-tunable foundation models: models that remain broadly usable in their released form while yielding limited adaptation gains under task-agnostic unauthorized fine-tuning. We propose Private Mask Pre-Training (PMP), a pre-training framework that concentrates representation learning into a sparse subnetwork identified early in training. The binary mask defining this subnetwork is kept private, and only the final dense weights are released. This forces unauthorized fine-tuning without access to the mask to update parameters misaligned with pretraining subspace, inducing an intrinsic mismatch between the fine-tuning objective and the pre-training geometry. We provide theoretical analysis showing that this mismatch destabilizes gradient-based adaptation and bounds fine-tuning gains. Empirical results on large language models demonstrating that PMP preserves base model performance while consistently degrading unauthorized fine-tuning across a wide range of downstream tasks, with the strength of non-fine-tunability controlled by the mask ratio.
