Search papers, labs, and topics across Lattice.
The paper introduces an action Jacobian penalty to discourage high-frequency control signals in reinforcement learning, addressing the issue of unnatural motions often produced by RL policies. To reduce the computational cost of this penalty, they propose a Linear Policy Net (LPN) architecture. Experiments demonstrate that LPNs with the action Jacobian penalty learn smooth policies faster, require no task-specific tuning, and are more efficient during inference, enabling successful motion imitation tasks and deployment on a physical quadrupedal robot.
Ditch hand-tuned action penalties: this work uses an action Jacobian penalty and a Linear Policy Net to automatically learn smooth, realistic robot control policies without task-specific tuning.
Reinforcement learning provides a framework for learning control policies that can reproduce diverse motions for simulated characters. However, such policies often exploit unnatural high-frequency signals that are unachievable by humans or physical robots, making them poor representations of real-world behaviors. Existing work addresses this issue by adding a reward term that penalizes a large change in actions over time. This term often requires substantial tuning efforts. We propose to use the action Jacobian penalty, which penalizes changes in action with respect to the changes in simulated state directly through auto differentiation. This effectively eliminates unrealistic high-frequency control signals without task specific tuning. While effective, the action Jacobian penalty introduces significant computational overhead when used with traditional fully connected neural network architectures. To mitigate this, we introduce a new architecture called a Linear Policy Net (LPN) that significantly reduces the computational burden for calculating the action Jacobian penalty during training. In addition, a LPN requires no parameter tuning, exhibits faster learning convergence compared to baseline methods, and can be more efficiently queried during inference time compared to a fully connected neural network. We demonstrate that a Linear Policy Net, combined with the action Jacobian penalty, is able to learn policies that generate smooth signals while solving a number of motion imitation tasks with different characteristics, including dynamic motions such as a backflip and various challenging parkour skills. Finally, we apply this approach to create policies for dynamic motions on a physical quadrupedal robot equipped with an arm.
