Track trials, findings, and clinical evidence as it lands.
Search papers, labs, and topics across Lattice.
This paper analyzes policy-gradient transfer from a simulator to a physical plant under a shared state estimator, where both systems are partially observed nonlinear with common Gaussian noise. We derive a novel bound on the policy gradient difference that is linear in noise-weighted model error and bypasses estimator accuracy requirements by leveraging the data-processing inequality. This bound enables an occupancy-weighted refinement, providing a sufficient condition for simulator gradients to be ascent directions for the plant and bounding expected squared plant gradients during finite simulator training.
A shared state estimator can enable effective policy-gradient transfer even with differing simulator and plant dynamics, by bounding the gradient discrepancy linearly with model error.
This paper considers the problem of policy-gradient transfer from a simulator to a physical plant when both are controlled through the same fixed state estimator. The plant and the simulator are partially observed nonlinear systems with additive Gaussian noise of common known covariance whose dynamics and observation models differ, and the controller is a squashed Gaussian policy with uniformly bounded latent mean and first two parameter derivatives. We derive a bound on the difference between the plant and simulator policy gradients that is linear in the noise-weighted model error and requires no estimator-accuracy guarantee, because the shared estimator is a common post-processing of the measurements and the data-processing inequality compares the trajectory laws without an estimation-error term. Based on this bound, we derive an occupancy-weighted refinement, a sufficient condition for the simulator gradient to be an ascent direction for the plant, and a bound on the expected squared plant gradient at a random iterate of a finite simulator-training run with optimization, truncation and model-error terms. Simulation results on a scalar system and a pendulum are presented to validate the theoretical results.