Search papers, labs, and topics across Lattice.
This paper critiques the conservative rollout strategies in reinforcement learning (RL) that limit exploration and behavioral diversity in embodied world models. It introduces a dual approach combining "Reward as an Agent," which provides reliable verification of behaviors to prevent reward hacking, and "Dynamic-Aware Rollout Diversification" (DynDiff-GRPO), which enhances action-space exploration. The results show that this unified framework not only mitigates reward hacking but also achieves significant accuracy improvements across various open-source world models, highlighting the potential for broader exploration grounded in robust verification.
Reward hacking can be effectively mitigated by a novel agentic reward framework that enhances exploration in RL, leading to substantial accuracy gains in embodied world models.
While RL has become a promising tool for refining world models, existing methods largely rely on conservative rollouts near the training distribution, limiting exploration, behavioral diversity, and richer dynamic discovery. In this work, we challenge this conservative paradigm. We argue that the core limitation is not exploration itself, but the lack of reliable verification strategies to support broader exploration. Without reliable verification, expanded exploration becomes highly susceptible to reward hacking, where policies exploit imperfect rewards without achieving genuine improvement. To evaluate this motivation, we instantiate our method in embodied world models, where physical plausibility, and task completion provide a rigorous testbed for scalable RL under complex dynamics. On the verification side, we introduce Reward as an Agent, an agentic reward framework that actively evaluates generated behaviors to provide robust reward signals and mitigate reward hacking under distribution shifts. On the exploration side, we introduce Dynamic-Aware Rollout Diversification through DynDiff-GRPO, which explicitly expands action-space exploration to diversify trajectories, broaden state-action coverage, and encourage richer embodied behaviors beyond conservative rollout regimes. By unifying Reward as an Agent with DynDiff-GRPO, we enable RL on a more reliable reward foundation with substantially diversified sampling, effectively mitigating reward hacking while yielding significant accuracy gains across multiple open-source world models, thereby demonstrating that broader exploration can scale successfully when grounded in robust verification.
