Search papers, labs, and topics across Lattice.
This study investigates the effects of sequential Direct Preference Optimization (DPO) on language model alignment, focusing on how different preference objectives interact during training. By analyzing four distinct preference settings, the authors reveal that the impact of later training on earlier preferences is not uniform; instead, it varies based on the relationship between objectives, signal strength, and training order. Key findings indicate that while some preferences may degrade, others can remain stable or even improve, challenging the assumption that later objectives uniformly degrade earlier learning.
Sequential preference optimization reveals a complex landscape where later training can enhance or degrade earlier preferences, depending on objective relationships.
Aligning language models with human preferences often requires optimising multiple behavioural objectives. A practical approach is to apply these objectives sequentially using preference optimisation methods such as Direct Preference Optimisation (DPO), but it remains unclear whether later training uniformly degrades preferences learned earlier or whether the effect depends on the relationship between objectives. We study sequential DPO across four preference settings covering distributional conflict, multi-attribute interaction, strong safety signal, and compatible response-quality objectives. Using Llama-3.1-8B-Instruct with LoRA adapters, we evaluate all objectives after every stage with a fixed base-model reference. We find that sequential DPO does not produce a single forgetting pattern; preference change ranges from partial degradation to stability, pair-level redistribution, or positive transfer depending on objective relationship, signal strength, and training order. Pair-level analysis using length-normalised policy margins shows that aggregate metrics can mask heterogeneous changes across preference pairs, whereas quartile decomposition reveals that high-confidence pairs can either degrade or improve depending on the setting. Mechanistic diagnostics show that Stage~2 gradients and adapter updates are near-orthogonal to the previous objective across all settings, providing little evidence that direct gradient opposition is the primary driver. These findings suggest that future sequential alignment pipelines should account for objective compatibility and signal strength, rather than assuming that later objectives affect earlier preferences uniformly.
