Search papers, labs, and topics across Lattice.
This paper introduces the Aggregation-Expectation-Transformation (AET) framework, which unifies Scalarized Expected Return (SER) and Expected Scalarized Return (ESR) paradigms in nonlinear multi-objective reinforcement learning (MORL). By bridging these historically fragmented optimization strategies, the authors propose AETDICE, an offline RL algorithm that employs DICE-style density-ratio estimation for sample-based optimization from static datasets. The key result demonstrates that AETDICE effectively captures complex trade-offs such as risk aversion and fairness, addressing challenges that previous methods could not resolve.
AETDICE resolves the long-standing divide in nonlinear multi-objective RL, enabling nuanced trade-off optimization that previous methods overlooked.
Optimizing nonlinear preferences in multi-objective reinforcement learning (MORL) is essential for capturing complex trade-offs like risk aversion or fairness. However, such non-linearity has historically bifurcated nonlinear MORL objectives into two distinct paradigms: Scalarized Expected Return (SER) and Expected Scalarized Return (ESR). While SER requires global-level optimization and ESR requires non-Markovian policies, leading to fragmented optimization strategies, we bridge this divide through the Aggregation-Expectation-Transformation (AET) framework. By unifying both criteria through a tripartite decomposition of scalarization, AET provides a principled foundation for general nonlinear MORL. Building on this framework, we propose AETDICE, a tractable offline RL algorithm for AET objectives. By utilizing DICE-style density-ratio estimation in an augmented state space, AETDICE enables sample-based optimization from static datasets. Our framework resolves long-standing barriers and captures respective trade-offs induced by AET framework, which existing methods fail to address.
