Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the discrepancies between reinforcement learning (RL) rewards and intended outcomes in the context of drag reduction in wall turbulence. By identifying issues with existing control methods that misrepresent performance, the authors introduce a corrected approach that utilizes a differentiable projection, a recurrent policy, and a more accurate reward structure. The result is a controller that achieves a conservative 17% improvement in drag reduction while adhering to a closed energy budget, addressing the pitfalls of reward hacking in RL applications.
Reward hacking can lead to misleading performance metrics, but a new approach restores accountability in drag reduction, achieving a 17% improvement under honest accounting.
A reinforcement-learning agent maximises its reward, which can diverge from the outcome its designer intended. In physical control the reward rarely closes that gap, and drag reduction in wall turbulence makes it concrete. A mass-conservation projection couples agents' outputs and erases the per-agent credit the policy gradient needs; a memoryless policy cannot resolve the slow near-wall cycle it acts on; and a pressure-gradient reward pays for nominal drag reduction by pumping power through the wall. Two degenerate controllers achieve large drag reductions while total dissipation rises, so the reported figure can mask a more wasteful flow. We trace each fault to its cause and fix it: a differentiable projection that restores credit, a recurrent policy with a widened sensing stencil, and a reward scored on the true wall power. The corrected controller acts on the flow within a closed energy budget, earning a conservative $17\%$ under honest accounting.