Search papers, labs, and topics across Lattice.
This paper introduces the Mean-Field-PhiBE (MF-PhiBE), a novel approach for continuous-time mean-field reinforcement learning that effectively utilizes discrete-time transition data to address the non-identifiability of drift and diffusion coefficients in McKean-Vlasov dynamics. By integrating discrete-time information into a continuous-time PDE framework on the Wasserstein space, the MF-PhiBE retains the generator structure of the Hamilton-Jacobi-Bellman equation while enabling a model-free actor-critic method. The authors demonstrate that the optimal policy derived from MF-PhiBE approximates the continuous-time value with an error of order \(螖t\), achieving second-order accuracy in linear-quadratic scenarios with minimal data requirements.
Bridging discrete-time data with continuous-time dynamics, the MF-PhiBE achieves unprecedented accuracy in mean-field reinforcement learning.
This paper addresses model-free continuous-time mean-field control in a setting where the population dynamics evolve continuously according to an unknown McKean-Vlasov stochastic differential equation, while only discrete-time transition data are available. In the model-based formulation, policy evaluation is naturally described by a stationary Hamilton-Jacobi-Bellman equation on $\mathcal P_2(\mathbb R^d)$, but this equation involves the drift and diffusion coefficients of the controlled McKean-Vlasov dynamics, which are not identifiable when only discrete-time data are available. On the other hand, a direct reduction to a time-discrete Bellman equation avoids the non-identifiability issue but loses the differential equation structure. To bridge these two viewpoints, we introduce a Mean-Field-PhiBE (MF-PhiBE), which incorporates discrete-time transition information into a continuous-time PDE on the Wasserstein space. The MF-PhiBE replaces the unknown infinitesimal drift and covariance in the policy-evaluation equation by one-step estimators computed from data, while preserving the generator structure of the McKean-Vlasov HJB equation. We also derive a policy-gradient theorem for entropy-regularized randomized feedback policies, expressing the actor direction through an action-wise infinitesimal advantage and the score of the policy. Combining these two ingredients yields a model-free actor-critic method. We prove a first-order consistency estimate showing that the value induced by an optimal MF-PhiBE policy approximates the optimal continuous-time value with an error of order $螖t$. In the linear-quadratic case, we show our approximation achieves second-order accuracy with only one-step data. Numerical experiments on an LQR benchmark and a crowd-aversion problem illustrate the proposed framework.
