Search papers, labs, and topics across Lattice.
This paper frames MDP planning as Bayesian inference over policies, assigning each policy an unnormalized probability of optimality based on its expected return. They approximate the resulting posterior distribution using a modified variational sequential Monte Carlo (VSMC) algorithm tailored for deterministic policies under stochastic dynamics, incorporating a sweep to ensure policy consistency and couple transition randomness. The approach uses posterior predictive sampling for action selection, effectively implementing a stochastic control policy through a Thompson-sampling lens.
Forget entropy regularization—this work shows how directly inferring a distribution over optimal policies in MDPs, using a modified VSMC, yields a stochastic control policy via Thompson sampling.
We cast episodic Markov decision process (MDP) planning as Bayesian inference over _policies_. A policy is treated as the latent variable and is assigned an unnormalized probability of optimality that is monotone in its expected return, yielding a posterior distribution whose modes coincide with return-maximizing solutions while posterior dispersion represents uncertainty over optimal behavior. To approximate this posterior in discrete domains, we adapt variational sequential Monte Carlo (VSMC) to inference over deterministic policies under stochastic dynamics, introducing a sweep that enforces policy consistency across revisited states and couples transition randomness across particles to avoid confounding from simulator noise. Acting is performed by posterior predictive sampling, which induces a stochastic control policy through a Thompson-sampling interpretation rather than entropy regularization. Across grid worlds, Blackjack, Triangle Tireworld, and Academic Advising, we analyze the structure of inferred policy distributions and compare the resulting behavior to discrete Soft Actor-Critic, highlighting qualitative and statistical differences that arise from policy-level uncertainty.
