Search papers, labs, and topics across Lattice.
This paper addresses the challenge of reinforcement learning in hybrid action spaces by proposing Hybrid TD3, an extension of TD3 tailored for such spaces. The work provides a theoretical analysis of overestimation bias in hybrid action settings, deriving bias bounds for twin-critic architectures and comparing five algorithmic variants. The authors introduce a weighted clipped Q-learning target to mitigate overestimation bias, leading to improved training stability and performance compared to existing hybrid action baselines.
Hybrid TD3 stabilizes reinforcement learning in hybrid action spaces by taming overestimation bias with a theoretically grounded weighted clipped Q-learning target.
Reinforcement learning in discrete-continuous hybrid action spaces presents fundamental challenges for robotic manipulation, where high-level task decisions and low-level joint-space execution must be jointly optimized. Existing approaches either discretize continuous components or relax discrete choices into continuous approximations, which suffer from scalability limitations and training instability in high-dimensional action spaces and under domain randomization. In this paper, we propose Hybrid TD3, an extension of Twin Delayed Deep Deterministic Policy Gradient (TD3) that natively handles parameterized hybrid action spaces in a principled manner. We conduct a rigorous theoretical analysis of overestimation bias in hybrid action settings, deriving formal bounds under twin-critic architectures and establishing a complete bias ordering across five algorithmic variants. Building on this analysis, we introduce a weighted clipped Q-learning target that marginalizes over the discrete action distribution, achieving equivalent bias reduction to standard clipped minimization while improving policy smoothness. Experimental results demonstrate that Hybrid TD3 achieves superior training stability and competitive performance against state-of-the-art hybrid action baselines
