Search papers, labs, and topics across Lattice.
This paper introduces a local geometric framework to analyze the fragility of alignment in fine-tuning language models, revealing that traditional first-order safety guarantees are misleading due to second-order effects in parameter-space trajectories. By formalizing the Alignment Instability Condition (AIC), the authors identify geometric properties that predict alignment degradation during fine-tuning, demonstrating a quartic onset of degradation linked to task coupling and parameter sensitivity. Empirical validation shows that the Fisher Information Matrix can effectively gauge safety degradation across various fine-tuning scenarios, providing a principled tool for understanding and mitigating alignment collapse.
Fine-tuning can lead to alignment collapse even when using benign tasks, with second-order effects proving more dangerous than previously understood.
Fine-tuning aligned language models on benign tasks (e.g. math tutoring) systematically breaks safety guardrails, even when training data contains no harmful content. While mechanistic approaches have shed light on where alignment resides in model weights, they do not by provide a general formal framework for deriving guarantees about when fine-tuning degrades it -- leaving the field without principled tools for predicting or preventing alignment collapse. We develop a local geometric framework through geometric analysis of parameter-space trajectories and apply it to understand the fragility of alignment in fine-tuning. While first-order analysis suggests orthogonal updates are safe, we prove this is illusory: the curvature of the fine-tuning loss induces second-order acceleration that can induce second-order drift into alignment-sensitive regions. We formalize a construct of our framework as the Alignment Instability Condition (AIC), three geometric properties that, when present, are sufficient to guarantee degradation. Our main result proves quartic onset of alignment degradation along gradient-flow trajectories, determined by how sharply alignment depends on specific parameters and how strongly tasks couple to these parameters. These findings yield formal sufficient conditions under which static first-order protection can fail under gradient descent. We further empirically validate the framework's foundations, showing that the Fisher Information Matrix provides a proxy for the degree of safety degradation across diverse fine-tuning.
