Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper evaluates reinforcement learning (RL) algorithms by emphasizing practical efficiency and robustness under dynamics mismatch, rather than solely focusing on sample efficiency. The authors find that the sample-inefficient Proximal Policy Optimization (PPO) algorithm can yield effective policies more quickly than sample-efficient alternatives like Soft Actor-Critic (SAC) and TD-MPC2, underscoring the significance of wall-clock time in training. Additionally, they demonstrate that domain randomization can enhance the robustness of policies across different RL paradigms, revealing a consistent impact on PPO, SAC, and TD-MPC2 when subjected to the same transfer protocol.
Sample efficiency isn't everything—PPO outperforms SAC and TD-MPC2 in training speed, challenging conventional wisdom in RL algorithm selection.
Transfer-oriented reinforcement learning requires evaluating algorithms along dimensions that go beyond standard sample efficiency. We focus on two dimensions: practical efficiency, which asks whether conclusions about algorithm suitability change under wall-clock rather than interaction-based budgets, and robustness under dynamics mismatch, which asks how different learning paradigms respond to variability in the training distribution induced by domain randomization. We provide two insights to reinforcement-learning practitioners. First, comparing the sample efficiency of different algorithms is often an insufficient criterion in transfer-oriented settings. The wall-clock time required to train a decent policy is an important consideration for practitioners, and we find that the sample-inefficient PPO algorithm can produce a performant policy faster than relatively more sample-efficient algorithms such as SAC and TD-MPC2, validating the common understanding of massively parallel training paradigms. Second, domain randomization can help different kinds of algorithms learn robust policies. In particular, although PPO, SAC, and TD-MPC2 represent different RL paradigms - on-policy, off-policy, and model-based learning and planning, respectively - we find that domain randomization affects all three algorithms in a similar way. To the best of our knowledge, this is the first controlled comparison of the effect of domain-randomization coverage on PPO, SAC, and TD-MPC2 under the same transfer protocol. Taken together, these two insights highlight the importance of evaluating RL algorithms not only by sample efficiency, but also by practical considerations such as training time and the algorithms' ability to produce usable policies.