Search papers, labs, and topics across Lattice.
This paper analyzes the impact of learning rates on PPO actor-critic methods by examining the behavior of hidden neurons using the Overfitting-Underfitting Indicator (OUI), a metric that quantifies the balance of binary activation patterns. They derive a theoretical connection between learning rate and activation sign changes, showing how neuron structure evolution depends on step size. Empirically, OUI measured at 10% of training effectively discriminates between different learning rate regimes and provides the strongest early screening signal compared to return, clip, divergence, and flip-based criteria, enabling aggressive pruning of unpromising runs.
Forget return curves – a simple measure of neuron activation patterns (OUI) at just 10% of training can predict PPO performance better than existing methods, enabling early pruning of bad runs.
Deep Reinforcement Learning systems are highly sensitive to the learning rate (LR), and selecting stable and performant training runs often requires extensive hyperparameter search. In Proximal Policy Optimization (PPO) actor--critic methods, small LR values lead to slow convergence, whereas large LR values may induce instability or collapse. We analyse this phenomenon from the behavior of the hidden neurons in the network using the Overfitting-Underfitting Indicator (OUI), a metric that quantifies the balance of binary activation patterns over a fixed probe batch. We introduce an efficient batch-based formulation of OUI and derive a theoretical connection between LR and activation sign changes, clarifying how a correct evolution of the neuron's inner structure depends on the step size. Empirically, across three discrete-control environments and multiple seeds, we show that OUI measured at only 10\% of training already discriminates between LR regimes. We observe a consistent asymmetry: critic networks achieving highest return operate in an intermediate OUI band (avoiding saturation), whereas actor networks achieving highest return exhibit comparatively high OUI values. We then compare OUI-based screening rules against early return, clip-based, divergence-based, and flip-based criteria under matched recall over successful runs. In this setting, OUI provides the strongest early screening signal: OUI alone achieves the best precision at broader recall, while combining early return with OUI yields the highest precision in best-performing screening regimes, enabling aggressive pruning of unpromising runs without requiring full training.
