Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper establishes the Environment Parameter Gradient Theorem, which provides a formal expression for the gradient of the value function concerning environment parameters in reinforcement learning (RL). By introducing a generalized action-value function that decouples the environment parameters governing current dynamics and future rollouts, the authors derive a tractable closed-form gradient expression. The proposed model-free algorithm effectively optimizes both the control policy and the environment parameters, demonstrating its utility in a UAV network design problem where optimal placement and communication routes are learned to minimize costs.
Jointly optimizing RL policies and environment parameters can significantly reduce operational costs in complex systems like UAV networks.
Reinforcement learning (RL) is traditionally concerned with learning a control policy for a fixed environment. In many engineering systems, however, the environment itself is alterable: physical or operational parameters can be tuned to shape the transition dynamics and costs experienced by the agent. This motivates jointly optimizing both the policy and the environment design parameters. To this end, we establish an Environment Parameter Gradient Theorem -- a formal expression for the gradient of the value function with respect to environment parameters. The key theoretical device is a generalized action-value function $Q_{π,ξ}(s,a,ζ)$, which comprises two copies of the environment parameters: $ζ$ governs the cost and transition dynamics at the current state--action pair, while $ξ$ governs the future rollouts. This decoupling yields a tractable closed-form gradient expression and is essential to the theorem's derivation. Building on this result, we develop a model-free algorithm that simultaneously learns the optimal policy and the environment parameters. We demonstrate the efficacy of our framework on a UAV network design problem, where the optimal UAV placement (environment parameters) and communication routes (governed by the policy) are learned jointly to minimize the total communication cost in the network.