Search papers, labs, and topics across Lattice.
This paper investigates the reliability of epistemic uncertainty quantification in latent dynamics models, specifically Recurrent State Space Models (RSSMs) used in the Dreamer family. It finds that latent transitions exhibit attractor behavior, biasing them towards well-represented regions of the latent space and causing discrepancies with actual environment dynamics. Consequently, epistemic uncertainty estimates become unreliable, and reward predictions are systematically overestimated, highlighting limitations in using these estimates for exploration and model exploitation mitigation.
Latent dynamics models like Dreamer can lure you into a false sense of security: their epistemic uncertainty estimates are unreliable because they're biased towards high-reward attractors in the latent space, even when the real world is different.
Model-Based Reinforcement Learning distinguishes between physical dynamics models operating on proprioceptive inputs and latent dynamics models operating on high-dimensional image observations. A prominent latent approach is the Recurrent State Space Model used in the Dreamer family. While epistemic uncertainty quantification to inform exploration and mitigate model exploitation is well established for physical dynamics models, its transfer to latent dynamics models has received limited scrutiny. We empirically demonstrate that latent transitions are biased toward well-represented regions of latent space, exhibiting an attractor behavior that can deviate from true environment dynamics. As a result, discrepancies in environment dynamics may not manifest in latent space, undermining the reliability of epistemic uncertainty estimates. Because these attractors often lie in high-reward regions, latent rollouts systematically overestimate predicted rewards. Our findings highlight key limitations of epistemic uncertainty estimation in latent dynamics models and motivate more critical evaluation of this method.
