Search papers, labs, and topics across Lattice.
This paper investigates how the measurement of entropy influences the policy geometry in Proximal Policy Optimization (PPO) for continuous control tasks. By analyzing an 80-muscle MyoLeg task, the authors reveal that the choice of entropy measurement significantly affects the distribution of actions near their bounds, with a clipped Gaussian achieving 89.07% of actions within 5% of the bound. The findings indicate that the relationship between mean and variance in policy design is complex, suggesting that entropy measurement is a critical factor in optimizing bounded policies beyond mere task performance metrics.
Where you measure entropy can drastically change the learned policy geometry, with implications for how we design continuous control systems.
Many continuous-control policies are optimized as unbounded Gaussians and then mapped into bounded actions. We show that where entropy is measured changes the policy geometry learned by proximal policy optimization (PPO). In an 80-muscle MyoLeg task, a clipped Gaussian executes 89.07% of actions within 5% of a bound. A same-state decomposition shows that this is not due to variance alone: setting variance to zero still leaves 83.83% of actions near a bound, while 82.12% of state-conditioned means lie outside the executable interval. Replacing clipping with a tanh map does not remove the high-variance regime. For latent Gaussian entropy H(u), the entropy loss has zero gradient with respect to the mean and a constant variance-increasing gradient. For executed-action entropy H(a), the transform Jacobian adds an inward gradient on the mean. Across three matched MyoLeg seeds, near-boundary occupancy is 71.42%, 29.76%, and 18.83% under latent entropy, no entropy, and executed-action entropy. A 38-dimensional Dog-Stand replication with an independent CleanRL-based PPO implementation reproduces the ordering in mean geometry, which also survives shared-state evaluation and boundary margins from 1% to 10%. Direct mean penalties can match or exceed the centering produced by H(a), showing that interior means are not unique to executed entropy. However, matched mean geometry can coexist with substantially different variance and return. Entropy measurement space is therefore a coupled mean-variance design choice, and task return alone does not characterize bounded-policy geometry.
