Search papers, labs, and topics across Lattice.
The paper introduces Log-Barrier Stochastic Gradient Bandit (LB-SGB), a novel policy optimization algorithm that incorporates a log-barrier regularization term to enforce exploration. This regularization ensures a minimal probability for all actions, addressing limitations of standard Stochastic Gradient Bandit (SGB) algorithms which assume the optimal action's probability is bounded away from zero. LB-SGB achieves comparable sample complexity to SGB while guaranteeing convergence even without assumptions on the learning process, and the authors demonstrate its effectiveness through simulations.
Log-barrier regularization can provably rescue policy optimization from getting stuck in suboptimal regions by structurally enforcing exploration, without sacrificing sample complexity.
Recently, it has been shown that the Stochastic Gradient Bandit (SGB) algorithm converges to a globally optimal policy with a constant learning rate. However, these guarantees rely on unrealistic assumptions about the learning process, namely that the probability of the optimal action is always bounded away from zero. We attribute this to the lack of an explicit exploration mechanism in SGB. To address these limitations, we propose to regularize the SGB objective with a log-barrier on the parametric policy, structurally enforcing a minimal amount of exploration. We prove that Log-Barrier Stochastic Gradient Bandit (LB-SGB) matches the sample complexity of SGB, but also converges (at a slower rate) without any assumptions on the learning process. We also show a connection between the log-barrier regularization and Natural Policy Gradient, as both exploit the geometry of the policy space by controlling the Fisher information. We validate our theoretical findings through numerical simulations, showing the benefits of the log-barrier regularization.
