Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper advances the understanding of coupled-dynamics environments by introducing optimal-control methods within the Joint Markov Decision Process (JMDP) framework, which retains dependencies across counterfactual outcomes. The authors define a nonparametric distributional Bellman optimality operator for JMDPs and demonstrate convergence of its iterates to the optimal joint return law under specific conditions. Key findings indicate that convergence can be achieved even when multiple mean-optimal actions exist, provided their tie resolutions share a second-moment fixed point, thus enhancing the potential for effective decision-making in complex environments.
The JMDP framework reveals that optimal control can be achieved even in environments with multiple competing actions, fundamentally shifting how we approach decision-making under uncertainty.
Coupled-dynamics environments expose the one-step outcomes that would follow from several possible counterfactual actions under a common realization of exogenous randomness. The ordinary Markov decision process formalism allows one to reason about the marginal law of each action but discards dependence across these counterfactual outcomes. The Joint Markov decision process (JMDP) formalism preserves that dependence. Prior work established the formalism and solved the fixed-policy joint moment evaluation problem in JMDPs. This paper develops optimal-control methods. We define a nonparametric distributional Bellman optimality operator for JMDPs, and prove that when the induced marginal MDP has a unique optimal policy, its iterates converge in Wasserstein distance to the optimal joint return law. For the first two moments, we establish convergence under a weaker condition that permits several mean-optimal actions as long as their tie resolutions share a second-moment fixed point. We also derive sampled targets for neural approximation.