Search papers, labs, and topics across Lattice.
This paper introduces an orthogonal bottleneck layer within deep RL agents to constrain encoder features to a low-dimensional subspace, aiming to improve representation learning without altering the underlying RL algorithm. They theoretically prove that expressivity is preserved when the bottleneck dimension exceeds the intrinsic rank of the optimal value function. Empirically, they demonstrate that this approach matches or improves baseline performance across various RL benchmarks, often with significant dimensionality reduction, and stabilizes feature norms.
RL agents can be drastically compressed into low-dimensional representations without sacrificing performance, challenging the need for high-dimensional feature spaces.
Deep reinforcement learning (RL) agents commonly rely on high-dimensional neural representations, despite growing evidence that task-relevant value and policy structure may be intrinsically low-dimensional. In this work, we present a simple yet effective representation-level prior that inserts a fixed orthonormal projection to constrain encoder features to a low-dimensional subspace, requiring no auxiliary objectives, pretraining, or changes to the underlying RL algorithm. Under a linear realizability assumption, we prove that when the bottleneck dimension exceeds the intrinsic rank of the optimal value function in feature space, the bottleneck preserves expressivity and leaves the induced gradient dynamics unchanged up to an equivalent low-dimensional parameterization. Empirically, we find that across both single and multi-task benchmarks, baseline performance is either matched or improved once the bottleneck dimension exceeds a small task-dependent threshold; in many cases, value representations can be compressed to extremely low dimensions without loss, and the minimal sufficient dimension depends far more on environment complexity than encoder width. In addition, we analyze representation geometry and find that orthogonal bottlenecks stabilize feature norms and are associated with higher effective rank. Together, these results support a representation-space interpretation of the manifold hypothesis in reinforcement learning and position orthogonal bottlenecks as a lightweight, architecture-agnostic mechanism for shaping RL representations.
