Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
The authors investigate whether policy gradient methods in robotic stabilization control require first-order (velocity) reward terms alongside zeroth-order (configuration) objectives. They theoretically and empirically establish that zeroth-order completeness over goal-relevant coordinates is sufficient, whereas adding first-order reward terms introduces severe hyperparameter sensitivity without aiding convergence, even though first-order states remain necessary in policy observations. This replaces pervasive, ad-hoc reward shaping heuristics borrowed from classical optimal control with a mathematically grounded blueprint for reward formulation.
Velocity penalties in RL reward functions are not just redundant for stabilization—they actively induce brittle policy gradient training as their coefficients scale.
Reward shaping is fundamental to modern robotic control with deep reinforcement learning (RL), yet practitioners still rely heavily on heuristic principles borrowed from classical optimal control and trajectory optimization. Existing methods rarely distinguish reward terms that are intrinsic to the control objective from numerical regularizers, leading to brittle hyperparameter tuning. To determine which quantities a reward must contain, we study the stabilization control problem with a focus on zeroth-order (configuration) and first-order (velocity) information. We theoretically and empirically demonstrate that policy gradient methods can successfully solve stabilization tasks without first-order reward terms, adding such terms can instead introduce severe sensitivity as their scale grows. Conversely, our findings confirm that reward functions must be zeroth-order complete over goal-relevant coordinates, while the first-order state remains necessary in the policy observation under our low-dissipation assumptions. Overall, these results provide actionable and principled guidance for reward design in robotic RL.