Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
This paper investigates the performance gap between single, state-augmented policies and multi-policy approaches in phase-structured, non-stationary RL environments. While theoretically a single policy can match multi-policy performance, practical limitations in function approximation and learning dynamics can favor multi-policy solutions. The authors propose a regime-based decomposition method, considering transient versus quasi-stationary dynamics, to predict which policy regime will yield better performance, validated through experiments on diverse non-stationary RL tasks.
Multi-policy RL can outperform single policies in non-stationary settings not due to theoretical limitations, but due to practical function approximation and learning dynamics, especially when phase durations are long or phases are highly heterogeneous.
Many reinforcement-learning (RL) problems are non-stationary yet structured and can be decomposed into phases, each with its own transition probabilities and reward functions. When the phase sequence is known, the common solution augments the state with information to satisfy the Markovian property and applies standard RL techniques. However, prior work finds that the multi-policy approach for different phases can outperform a single state-augmented policy shared among the phases, for reasons that remain unclear. In this work, we first show that the shared policy can theoretically achieve performance of any multi-policy solution. However, whether a multi-policy solution can perform better than the corresponding single shared policy in practice depends on function approximation, learning and optimization processes, as well as, for multi-policy solutions, the sample efficiency and loss of continuity from one policy to another. We propose a regime-based phase decomposition method to identify which policy can provide better performance. The method is based on consideration of the duration of transient system dynamics relative to the duration of the quasi-stationary period. Numerical experiments are conducted with different non-stationary RL problems to validate our four major hypotheses: (a) longer phase durations favor multi-policies, (b) the heterogeneity between phases increases the burden on single policy, (c) multi-policies need sufficient data for each phase, and (d) environment-specific transition dynamics between phases can affect which policy is preferable.