Search papers, labs, and topics across Lattice.
This paper introduces LP-SFT, a novel fine-tuning method that preserves the multimodal entropy structure of pretrained language models while adapting them to specific downstream tasks. By analyzing next-token predictions through Shannon and Renyi entropies, the authors demonstrate that standard supervised fine-tuning can distort the rich distributional knowledge encoded in the model, leading to capability degradation. LP-SFT effectively maintains this structure by applying a locally normalized preservation loss, resulting in improved performance metrics across various fine-tuning scenarios compared to traditional methods.
Local-Preserving Supervised Fine-Tuning can enhance model performance without sacrificing the rich diversity of pretrained knowledge, achieving superior results in both accuracy and diversity metrics.
Supervised fine-tuning (SFT) is the standard approach for adapting pretrained language models to downstream domains, yet it often improves target-domain behavior at the cost of degrading pre-existing capabilities. Standard cross-entropy fine-tuning promotes only the observed label token and leaves unconstrained how probability mass is redistributed over other plausible alternatives, potentially distorting the rich local preference structure learned during pretraining. We first analyze next-token predictions using Shannon and Renyi entropies, revealing that pretrained models exhibit a regular multimodal entropy structure. These entropy peaks correspond to varying numbers of plausible alternatives, indicating that the base model intrinsically encodes rich distributional knowledge beyond the single supervised token. Motivated by this observation, we propose LP-SFT, a Local-Preserving Supervised Fine-Tuning objective designed to explicitly protect this inherent entropy structure. At each step, LP-SFT constructs an adaptive support of alternative tokens and applies a locally normalized preservation loss to maintain the base model's relative structure among them, while standard cross-entropy independently optimizes the supervised token. Across mixed-domain and single-domain fine-tuning experiments, LP-SFT improves overall performance over vanilla SFT and recent SFT-enhancement baselines, achieving the best balance between pass@1 accuracy and pass@k performance. These results suggest that local preservation helps mitigate capability degradation without collapsing sampling-accessible diversity.
