Search papers, labs, and topics across Lattice.
This paper introduces a method for dynamically adjusting the granularity of state-action abstractions in reinforcement learning based on a novel performance certificate that balances learning error and abstraction error. By employing soft state-action abstractions derived from rate-distortion principles, the authors demonstrate that their approach allows for continuous resolution adjustments, leading to efficient learning even under significant lossy compression. Experimental results across various tabular settings reveal that this strategy can achieve near-optimal performance, highlighting its effectiveness in complex task environments.
Refining state-action abstractions based on learning error can lead to near-optimal performance even with substantial information loss.
When learning to walk, infants seem to address a coarse version of the problem first - stay upright, reach the caregiver - and refine it only when further practice at that resolution stops paying off. Reinforcement learning offers multiple techniques for building simple versions of complex tasks, but lacks general principles for how to dynamically adjust the granularity of these abstractions during learning. This paper proposes one such principle: refine the abstraction as soon as the learning error within it becomes comparable to the error induced by the abstraction itself. Here, we investigate one way of formalising this principle via a performance certificate that decomposes value error into two terms: a learning error bound captured by a Bellman residual, and an abstraction error bound given by a bisimulation metric. The resulting switching strategy is implemented by soft state-action abstractions built from rate-distortion principles, whose resolution along state and action axes can be continuously adjusted. We validate this construction in a range of tabular settings, showing that near-optimal performance can be achieved under substantial lossy compression of state and action information.
