Search papers, labs, and topics across Lattice.
This paper tackles the computational challenges of inverse reinforcement learning (IRL) by reformulating it as a bilevel optimization problem, where the outer level measures the discrepancy between the learned policy and expert demonstrations. The authors derive a structured Fisher-based hypergradient that leverages the relationship between the Hessian of the inner objective and the Fisher information matrix, enabling efficient updates without the need for explicit Fisher matrix construction. Their method, which employs a streaming spectral sketch to approximate the inverse-Fisher-vector product, shows competitive performance in both discrete and continuous control environments, significantly improving computational efficiency.
By leveraging a structured Fisher-based hypergradient, this approach reduces the complexity of inverse reinforcement learning, achieving competitive policy performance without the heavy computational burden of traditional methods.
Inverse reinforcement learning (IRL) aims to recover a reward function under which the resulting policy reproduces the behavior observed in expert demonstrations. A natural approach is to formulate IRL as a bilevel optimization problem, in which the inner level corresponds to policy optimization under the learned reward and the outer level measures the discrepancy between the induced policy and expert data. However, this formulation is computationally challenging in practice because the outer update requires a hypergradient involving an inverse-Hessian-vector product for the inner objective. We address this challenge by showing that, at the inner optimum, the Hessian of the inner objective is proportional to the Fisher information matrix of the policy, yielding a structured Fisher-based hypergradient closely related to Natural Hypergradient Descent. To address the resulting scalability bottleneck associated with large Fisher matrices, we approximate the required inverse-Fisher-vector product using a streaming spectral sketch, avoiding explicit construction of the Fisher matrix. We evaluate our approach against a first-order stochastic bilevel baseline across discrete- and continuous-control environments. The results demonstrate competitive policy performance and strong reward-ranking quality, while Fisher sketching reduces curvature-storage complexity and can improve computational efficiency relative to an explicit Fisher solver.
