Search papers, labs, and topics across Lattice.
This paper reformulates Active Inference (AIF) as a convex Markov decision process (MDP) to unify epistemic and pragmatic objectives through expected free energy (EFE) minimization. By establishing that the pragmatic components are linear in predictive state marginals, the authors demonstrate that EFE minimization can be approached similarly to reward maximization in latent MDPs, while the epistemic value introduces a critical nonlinear aspect. The study introduces a mirror descent algorithm that aligns AIF with actor-critic methods, suggesting a novel integration of world-model learning and policy optimization that enhances convergence and policy improvement guarantees.
Active Inference can be framed as a convex MDP, revealing a surprising connection to performative reinforcement learning with robust policy improvement guarantees.
Active Inference (AIF) frames adaptive behavior as the minimization of expected free energy (EFE), combining epistemic and pragmatic objectives within a single variational principle. We frame AIF as policy optimization and show that, for closed-loop control policies, EFE minimization can be formulated as a convex Markov decision process (MDP). In this formulation, the pragmatic terms are linear in the predictive state marginals and therefore equivalent to reward maximization in a latent MDP, while the epistemic value introduces a nonlinear component that distinguishes EFE minimization from standard reinforcement learning. This perspective further reveals the epistemic drive of active inference as a policy-dependent (performative) reward. We analyze finite-horizon, discounted, and average-reward formulations of EFE and derive a mirror descent (MD) algorithm that locally linearizes the objective around the current state marginals, yielding a policy-dependent reward that is compatible with actor-critic methods and dynamic programming. Finally, we argue that coupling world-model learning with policy optimization gives active inference the structure of performative reinforcement learning, providing a route toward grounding active inference within modern reinforcement learning and optimization theory, including convergence analysis and principled policy improvement guarantees.
