Search papers, labs, and topics across Lattice.
This paper introduces the concept of critic complexity as a diagnostic tool in actor-critic reinforcement learning, utilizing spectral effective-rank entropy to quantify the complexity of critic models. Through experiments with TD3 and PPO, the authors establish a measurable link between critic complexity and training behavior, revealing that this relationship varies significantly across different algorithms and tasks. Furthermore, they demonstrate that by incorporating a spectral-entropy penalty into the critic loss, it is possible to control critic complexity, thereby influencing training dynamics and outcomes.
Critic complexity can be directly controlled in reinforcement learning, offering a new lever for optimizing training performance.
Actor-critic methods depend on learned critics, but critic quality is often evaluated only indirectly through return, temporal-difference error, or value loss. Critic complexity is introduced as an additional diagnostic and intervention dimension for actor-critic reinforcement learning. The analysis uses spectral effective-rank entropy, a rank-like summary of the singular-value distributions of critic weight matrices, to assess critic model complexity. Across TD3 and PPO experiments, critic complexity is tracked together with return and Monte Carlo value-estimation bias. The results show that critic complexity is measurable throughout training and is systematically associated with training behavior, while also making clear that the relationship is heterogeneous across algorithms, tasks, and hyperparameters. A direct complexity-control intervention is then evaluated by adding a spectral-entropy penalty to the critic loss. This intervention reliably changes the targeted spectral quantity, demonstrating that critic complexity can be controlled rather than only observed. Return effects are treated as task-dependent evidence rather than as a general performance claim, because overall complexity-control results vary.
