Search papers, labs, and topics across Lattice.
This paper introduces a continuous-time model-free reinforcement learning algorithm designed to learn deterministic equilibrium policies in time-inconsistent control problems. By reformulating the problem into a two-stage process using the extended Hamilton-Jacobi-Bellman system, the authors successfully apply a deterministic policy gradient approach to derive optimal policies and utilize fixed point iterations to refine auxiliary functions. The algorithm demonstrates significant effectiveness in two financial applications, showcasing its potential to address time-inconsistency in decision-making frameworks.
Time-inconsistent control problems can be tackled effectively with a novel two-stage reinforcement learning algorithm that learns equilibrium policies, bridging theory and practical applications in finance.
In this paper, we develop a continuous-time model-free reinforcement learning algorithm to learn deterministic equilibrium policies in general time-inconsistent control problems. Utilizing the extended Hamilton-Jacobi-Bellman system, we recast the original time-inconsistent problem into an equivalent two-stage problem. In the first stage, for given auxiliary functions, we employ the deterministic policy gradient approach to learn an optimal policy in an auxiliary time-consistent control problem. In the second stage, given the updated policy, we exploit the inner fixed point iterations and some martingale characterizations to learn the auxiliary functions. As a theoretical contribution, we provide some mild model assumptions and establish the convergence of inner fixed point iterations. By repeating this actor-critic style of iterations across two stages, our algorithm aims to learn the equilibrium under different sources of time-inconsistency in a unified manner. The superior effectiveness of the proposed algorithm are illustrated in two classical financial applications with time-inconsistency: mean-variance portfolio management and optimal tracking portfolio under non-exponential discounting.
