Search papers, labs, and topics across Lattice.
This paper introduces LAC (Light Actor, deep Critic), an offline reinforcement learning method that reallocates model capacity from a complex actor to a more powerful critic, enhancing inference efficiency. By addressing the instabilities associated with deeper critics鈥攕uch as optimization challenges, bootstrap-noise amplification, and value-range drift鈥攖hrough innovative architectural and training strategies, LAC achieves significant performance improvements. The method matches the performance of advanced generative actors while reducing inference latency by up to four times, demonstrating a practical path for scalable RL deployment.
By shifting capacity from complex actors to deep critics, LAC achieves state-of-the-art performance with dramatically lower inference latency.
Recent progress in offline reinforcement learning (RL) has been driven by expressive generative actors such as diffusion and flow-matching policies, which capture multimodal behavior in offline datasets. However, these actors require multiple denoising or integration steps per action and thus incur substantial overhead at every decision in deployment. In this work, we revisit where capacity should be invested in an offline actor--critic method. Since the critic is used only during training and is discarded at deployment while the actor runs at every decision step, allocating capacity to the critic rather than the actor is more favorable for inference-time efficiency. However, scaling MLP critics in offline RL is known to introduce several distinct instabilities that have, in practice, kept critics shallow. We identify three distinct failure modes that arise when critics are deepened in offline RL---optimization, bootstrap-noise amplification, and value-range drift---and address each with a corresponding ingredient: a residual MLP backbone, n-step bootstrap targets, and a categorical cross-entropy loss. Combining these ingredients with a lightweight deterministic actor, we propose LAC (Light Actor, deep Critic). On OGBench, LAC matches the strongest diffusion- and flow-matching baselines while achieving up to 4x lower inference latency, comparable to one-step distilled policies without distillation. Its critic recipe also transfers across actor parametrizations.
