Search papers, labs, and topics across Lattice.
The paper introduces Pareto Multi-Objective Alignment (PAMA), a novel algorithm for aligning LLMs with multiple, potentially conflicting objectives, addressing the limitations of single-reward RLHF approaches. PAMA reformulates multi-objective RLHF as a convex optimization problem with a closed-form solution, achieving O(n) complexity (where n is the number of objectives) and enabling efficient optimization. Experiments on LLMs (125M-7B parameters) demonstrate PAMA's effectiveness in achieving Pareto stationary points, balancing objectives like informativeness, conciseness, helpfulness, and creativity.
Aligning LLMs to multiple objectives just got 1000x faster: PAMA reduces the computational complexity of multi-objective RLHF from quadratic to linear, enabling practical Pareto optimization.
Large language models (LLMs) are increasingly deployed in real-world applications that require careful balancing of multiple, often conflicting, objectives, such as informativeness versus conciseness, or helpfulness versus creativity. However, current alignment methods, primarily based on RLHF, optimize LLMs toward a single reward function, resulting in rigid behavior that fails to capture the complexity and diversity of human preferences. This limitation hinders the adaptability of LLMs to practical scenarios, making multi-objective alignment (MOA) a critical yet underexplored area. To bridge this gap, we propose Pareto Multi-Objective Alignment (PAMA), a principled and computationally efficient algorithm designed explicitly for MOA in LLMs. In contrast to computationally prohibitive multi-objective optimization (MOO) methods, PAMA transforms multi-objective RLHF into a convex optimization with a closed-form solution, significantly enhancing scalability. Traditional MOO approaches suffer from prohibitive O(n^2*d) complexity, where d represents the number of model parameters, typically in the billions for LLMs, rendering direct optimization infeasible. PAMA reduces this complexity to O(n) where n is the number of objectives, enabling optimization to be completed within milliseconds. We provide theoretical guarantees that PAMA converges to a Pareto stationary point, where no objective can be improved without degrading at least one other. Extensive experiments across language models ranging from 125M to 7B parameters demonstrate PAMA's robust and effective MOA capabilities, aligning with its theoretical advantages. PAMA provides a highly efficient solution to the MOA problem that was previously considered intractable, offering a practical and theoretically grounded approach to aligning LLMs with diverse human values, paving the way for versatile and adaptable real-world AI deployments.
