Search papers, labs, and topics across Lattice.
This paper introduces a novel method, \methodname, for multi-preference alignment in reinforcement learning from human feedback (RLHF) that optimizes the timing of when each preference dimension is incorporated into policy optimization. By employing a stability-guided active-set controller, \methodname dynamically adjusts the active set of objectives based on performance metrics, leading to improved average performance across multiple training preferences compared to traditional scalarization methods. The findings demonstrate that effective management of objective admission can significantly enhance alignment outcomes in complex multi-objective scenarios.
Timing the entry of preference dimensions can lead to substantial performance gains in multi-preference alignment for LLMs.
Multi-preference alignment is often framed as scalarization: combine reward dimensions, then optimize. This leaves a temporal decision underspecified: when should each preference dimension enter policy optimization? We propose \methodname, a stability-guided active-set controller for controlled objective admission. \methodname starts from a small active set, retains admitted objectives, and expands when reward-deviation gates indicate low recent deviation or a patience budget is exhausted. A probing phase estimates a hard-to-easy order, and adaptive weighting emphasizes underperforming active dimensions. Automatic evaluations with 15 training preferences and 16 held-out benchmark columns show that \methodname obtains higher averages than simultaneous scalarization and shared-budget adapted baselines. Component ablations and expansion dynamics further support cumulative retention, gated admission, and probing-derived ordering as useful design choices in this setting. These results position objective-entry timing as a concrete control variable in reward-vector RLHF.
