Search papers, labs, and topics across Lattice.
This paper critiques the one-size-fits-all approach in online reinforcement learning (RL) that relies on offline priors, highlighting the variability of prior validity across different deployments. The authors advocate for a diagnosis-driven tension management framework that uses deployment-specific evidence to inform how RL agents interact with their priors during training, rather than adhering to universal solutions. Key findings include controlled experiments showing that the effectiveness of priors can reverse depending on context, underscoring the need for adaptive strategies in RL deployment.
Prior validity in online RL can shift dramatically, making universal solutions ineffective and necessitating a tailored, evidence-driven approach for each deployment.
Online reinforcement learning (RL) agents increasingly depend on knowledge acquired offline to achieve practical efficiency. Originally studied in offline-to-online RL, this paradigm now spans foundation model post-training and embodied intelligence, with prior types expanding from offline datasets and pre-trained policies to increasingly diverse knowledge sources such as multimodal foundation models and generative world models. Offline priors have become central to how deep RL is developed and deployed. However, this reliance introduces a challenge that the prevailing benchmark-driven paradigm cannot resolve: because prior validity varies across deployments and shifts during training, no single approach to managing it is universally optimal, and benchmark rankings offer limited guidance for real-world deployments. Rather than pursuing universal solutions, we argue that the field should shift to diagnosis-driven tension management, in which deployment-specific evidence guides how the learner relates to its priors throughout training, enabling both flexible and adaptive deployment. We support this position with a framework characterizing how priors reshape online optimization through three functional roles, controlled experiments demonstrating help-or-hurt reversals, cross-domain evidence from foundation model post-training to embodied intelligence, and engagement with five substantive counterarguments.
