Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the challenges of applying Reinforcement Learning (RL) to chaotic dynamical systems, where sensitivity to initial conditions leads to high-variance bootstrap targets. It demonstrates that the return distribution, when measured using the 1-Wasserstein metric, exhibits more regular behavior than individual trajectories. The authors propose that distributional RL, by optimizing the distributional Bellman objective, provides better-conditioned learning in these systems, offering a theoretical explanation for its advantages.
Distributional RL tames the chaos of RL in dynamical systems, revealing that return distributions evolve more predictably than individual trajectories.
Chaotic dynamical systems pose a fundamental challenge for Reinforcement Learning (RL): exponential sensitivity to initial conditions induces high-variance bootstrap targets and poorly conditioned gradient updates. Chaotic dynamics arise across scientific and engineering domains, from fluid flows and climate systems to multi-agent systems, where reliable learning is highly desirable. Standard RL methods optimise expected returns through scalar value functions, implicitly averaging over diverging trajectories and entangling trajectory level instability with the learning objective. We show that under mild statistical stability assumptions, the return distribution evolves more regularly than individual trajectories when measured under the $1$-Wasserstein metric, yielding a smoother distributional Bellman objective. By aligning optimisation with this measure level structure, distributional RL provides better conditioned learning. We offer a principled explanation for the advantages of distributional methods in chaotic systems and the geometries of RL objectives under chaos.