Search papers, labs, and topics across Lattice.
FlashSAC, a Soft Actor-Critic variant, accelerates off-policy RL by reducing gradient updates and compensating with larger models and higher data throughput, drawing inspiration from supervised learning scaling laws. To address the instability introduced by this scaling, FlashSAC employs explicit weight, feature, and gradient norm bounding to control critic error accumulation. Empirical results across a diverse set of robot control tasks demonstrate that FlashSAC achieves superior performance and training efficiency compared to PPO and other off-policy methods, particularly in high-dimensional and sim-to-real scenarios.
Off-policy RL can now train in minutes instead of hours for sim-to-real robot control, thanks to a new algorithm that borrows scaling laws from supervised learning.
Reinforcement learning (RL) is a core approach for robot control when expert demonstrations are unavailable. On-policy methods such as Proximal Policy Optimization (PPO) are widely used for their stability, but their reliance on narrowly distributed on-policy data limits accurate policy evaluation in high-dimensional state and action spaces. Off-policy methods can overcome this limitation by learning from a broader state-action distribution, yet suffer from slow convergence and instability, as fitting a value function over diverse data requires many gradient updates, causing critic errors to accumulate through bootstrapping. We present FlashSAC, a fast and stable off-policy RL algorithm built on Soft Actor-Critic. Motivated by scaling laws observed in supervised learning, FlashSAC sharply reduces gradient updates while compensating with larger models and higher data throughput. To maintain stability at increased scale, FlashSAC explicitly bounds weight, feature, and gradient norms, curbing critic error accumulation. Across over 60 tasks in 10 simulators, FlashSAC consistently outperforms PPO and strong off-policy baselines in both final performance and training efficiency, with the largest gains on high-dimensional tasks such as dexterous manipulation. In sim-to-real humanoid locomotion, FlashSAC reduces training time from hours to minutes, demonstrating the promise of off-policy RL for sim-to-real transfer.
