Search papers, labs, and topics across Lattice.
This paper addresses the performance gap between PPO and SAC for legged robot locomotion by identifying and mitigating key factors hindering SAC's scalability. They introduce modifications to SAC, including policy initialization strategies, timeout-aware critic targets, and multi-step return estimation. The resulting SAC implementation matches PPO's performance across various legged robot platforms and locomotion tasks, making it a viable alternative for sim-to-real transfer.
SAC can now rival PPO in massively parallel legged robot training, unlocking its potential for efficient sim-to-real transfer and online adaptation.
Proximal Policy Optimization (PPO) has become the de facto standard for training legged robots, thanks to its robustness and scalability in massively parallel simulation environments like IsaacLab. However, its on-policy nature makes it inherently sample-inefficient, preventing its use for continuous adaptation and fine-tuning on real hardware. Soft Actor-Critic (SAC), by contrast, is an off-policy algorithm that can reuse past experience, making it a natural candidate for sim-to-real transfer workflows where the same algorithm can be used both in simulation and for online learning on the real robot. Despite these advantages, SAC has consistently failed to match PPO's empirical performance in massively parallel training settings. This work identifies the root causes of this gap and introduces targeted modifications, covering policy initialization, timeout-aware critic targets, and multi-step return estimation, that enable SAC to train stably at scale. Evaluated across multiple legged robot platforms and diverse locomotion tasks, our approach closes the performance gap with PPO entirely.
ETH