Search papers, labs, and topics across Lattice.
This paper reveals that classical world models, despite increased memory, cannot achieve perfect alignment with the true world in complex environments, as they inevitably fail to distinguish optimal actions or misassign rewards. In contrast, the authors demonstrate that a quantum world model using a single qutrit can perfectly replicate the true world, maintaining accurate reward estimates and aligning optimal policies. This finding underscores a fundamental limitation of classical approaches and highlights the potential of quantum models in achieving precise alignment in agent training.
Classical world models can fail to align with reality, but a quantum approach using a single qutrit achieves perfect alignment and optimal policy matching.
World models provide digital simulacra of the true world, allowing agents to be trained and tested before costly real-world deployment. At each time step, they receive an action and generate an observation and reward matching the statistics of the true world. In complex environments where present outcomes depend on events far in the past, this requires memory. One might expect that, by increasing memory, we can always build a model accurately enough to align the optimal agent policies of the real and virtual worlds. We show that this is false for classical world models, even when the true world itself is classical. We construct true worlds for which every finite classical model fails along the same possible trajectory: it either loses the ability to distinguish actions when the true world clearly prefers one, or repeatedly assigns the highest expected reward to suboptimal actions. Its expected-reward estimates also retain a nonvanishing average error. In contrast, each such true world admits a quantum world model using a single qutrit that reproduces it exactly: its reward estimates and preferred actions always match those of the true world, ensuring that the optimal policies of the real and virtual worlds remain perfectly aligned.
