Search papers, labs, and topics across Lattice.
This paper introduces Posterior Optimization with Clipped Objective (POCO), a novel RL framework for fine-tuning generative policies by framing policy improvement as posterior inference over temporal action chunks. POCO uses an Expectation-Maximization procedure to distill a reward-weighted implicit posterior into the policy, avoiding explicit likelihood estimation and improving stability. Empirical results across simulation and real-world robotic manipulation tasks show that POCO outperforms state-of-the-art baselines and prevents catastrophic policy collapse, achieving a 96.7% success rate on real-world tasks.
Fine-tuning generative policies for robotics doesn't have to be a nightmare: POCO offers a stable and efficient RL framework that actually works on real robots.
Expressive generative models have advanced robotic manipulation by capturing complex, multi-modal action distributions over temporally extended trajectories. However, fine-tuning these policies via RL remains challenging due to instability and sample inefficiency. We introduce Posterior Optimization with Clipped Objective (POCO), a principled RL framework that formulates policy improvement as a posterior inference problem tailored for temporal action chunks. Through an Expectation-Maximization procedure, POCO distills a reward-weighted implicit posterior into the policy without likelihood estimation. Furthermore, POCO adopts an offline-to-online paradigm that anchors online exploration to pre-trained priors, and its model-agnostic design scales to fine-tune large VLA models without architectural modifications. Evaluations across 7 simulation benchmarks and 4 contact-rich real-world tasks demonstrate that POCO prevents catastrophic policy collapse, outperforms SOTA baselines, and achieves a 96.7% success rate on real-world tasks. Videos are available at our project website https://cccedric.github.io/poco/.
