Search papers, labs, and topics across Lattice.
To eliminate metric oscillation and gradient tug-of-war in multi-objective reinforcement learning (MORL), the authors propose Multi-Marginal Preference Optimization (MMPO), an alternative to traditional linear reward scalarization. The framework mitigates optimization conflicts by integrating data-level exposure debiasing, priority-aware orthogonal gradient projections, and self-prompted gradient constraints to prevent dominant objectives from suppressing weaker ones. Evaluations on real-world e-commerce benchmarks, ToolRL, and code generation show that MMPO stabilizes training dynamics and achieves consistent Pareto gains across conflicting metrics.
Naive linear reward scalarization inevitably triggers destructive gradient tug-of-wars, but decoupling marginal objectives via orthogonal projections and localized constraints stabilizes training across both recommender systems and complex LLM alignment tasks.
Real-world Multi-Objective Reinforcement Learning (MORL) often suffers from sparse rewards, reward conflicts, and late-stage reward tug-of-war, causing traditional linear scalarization to experience severe metric oscillations. To address optimization conflicts among multiple objectives in real-world deployment scenarios, we propose Multi-Marginal Preference Optimization (MMPO), a fine-grained framework that intervenes at the data, gradient, and constraint levels rather than relying on coarse-grained global scalarization. Specifically, MMPO performs exposure debiasing to mitigate sparse and biased rewards, applies priority-aware orthogonal projection to decouple conflicting gradients, and introduces self-prompted gradient constraints to prevent dominant objectives from overwhelming weaker ones. Experiments on real-world e-commerce datasets show that MMPO improves training stability and consistently achieves better performance across conflicting metrics. Moreover, it generalizes robustly to broader tasks such as ToolRL and code generation, demonstrating its effectiveness as a practical paradigm for multi-objective alignment.
