Search papers, labs, and topics across Lattice.
This study conducts an extensive empirical analysis of actor-critic algorithms by evaluating over 33,000 experiments on a control task derived from a real water treatment plant. The research identifies critical design decisions that influence the reliability and hyperparameter sensitivity of these algorithms, revealing that common defaults like Gaussian action distributions are among the least reliable. In contrast, the use of bounded distributions with adaptive update schedules demonstrates robustness across diverse settings, providing valuable insights for practitioners in real-world applications.
Common defaults in actor-critic algorithms can lead to unreliable performance, while bounded distributions with adaptive updates prove to be significantly more robust.
Reinforcement learning is increasingly being considered for controlling real-world systems, from fusion plasma and autonomous vehicles to drug discovery and drinking water treatment, where reliability is essential and tuning budgets are limited. Actor-critic algorithms share a set of design decisions, such as how the policy is updated, how it represents the distribution over actions, how its gradient is estimated, and how often it is updated relative to the value estimator. Using a control task derived from a real water treatment plant, we analyze over 33,000 experiments to determine how these components affect variability across runs and sensitivity to hyperparameters. Common defaults, such as Gaussian action distributions with pathwise gradient estimators, are among the least reliable configurations, whereas bounded distributions with adaptive update schedules remain robust across a wide range of settings. These findings offer empirical guidance to practitioners across scientific and engineering domains for understanding and making component-level decisions when adapting actor-critic methods to new real-world control settings.
