Search papers, labs, and topics across Lattice.
SPAARS addresses the challenge of safe online exploration in offline-to-online RL by initially constraining exploration within a low-dimensional latent manifold learned via a conditional variational autoencoder (CVAE), and then transferring control to the raw action space to bypass the decoder's reconstruction bottleneck. The method is instantiated in two variants: one using unordered (s,a) pairs with a CVAE, and another combining SPAARS with OPAL temporal skill pretraining for structured exploration. Theoretical analysis provides an upper bound on the exploitation gap and demonstrates variance reduction in latent-space policy gradients, while empirical results show SPAARS achieves superior performance and sample efficiency compared to baselines on benchmark tasks.
Escape the CVAE decoder bottleneck: SPAARS unlocks better offline-to-online RL by safely exploring a latent space, then seamlessly switching to raw actions.
Offline-to-online reinforcement learning (RL) offers a promising paradigm for robotics by pre-training policies on safe, offline demonstrations and fine-tuning them via online interaction. However, a fundamental challenge remains: how to safely explore online without deviating from the behavioral support of the offline data? While recent methods leverage conditional variational autoencoders (CVAEs) to bound exploration within a latent space, they inherently suffer from an exploitation gap -- a performance ceiling imposed by the decoder's reconstruction loss. We introduce SPAARS, a curriculum learning framework that initially constrains exploration to the low-dimensional latent manifold for sample-efficient, safe behavioral improvement, then seamlessly transfers control to the raw action space, bypassing the decoder bottleneck. SPAARS has two instantiations: the CVAE-based variant requires only unordered (s,a) pairs and no trajectory segmentation; SPAARS-SUPE pairs SPAARS with OPAL temporal skill pretraining for stronger exploration structure at the cost of requiring trajectory chunks. We prove an upper bound on the exploitation gap using the Performance Difference Lemma, establish that latent-space policy gradients achieve provable variance reduction over raw-space exploration, and show that concurrent behavioral cloning during the latent phase directly controls curriculum transition stability. Empirically, SPAARS-SUPE achieves 0.825 normalized return on kitchen-mixed-v0 versus 0.75 for SUPE, with 5x better sample efficiency; standalone SPAARS achieves 92.7 and 102.9 normalized return on hopper-medium-v2 and walker2d-medium-v2 respectively, surpassing IQL baselines of 66.3 and 78.3 respectively, confirming the utility of the unordered-pair CVAE instantiation.
