Search papers, labs, and topics across Lattice.
This paper critiques the reliance on World Models (WMs) for evaluating action policies in robotics, emphasizing the need for rigorous validation of these models before trusting their verdicts. It introduces an admissibility ladder framework that categorizes the certification of WMs, drawing from established safety practices to ensure their reliability as test oracles. Notably, the study finds that higher visual fidelity in WMs does not correlate with better action-following performance, highlighting a critical gap in current evaluation methods.
Visual fidelity in World Models can be misleading; a model that looks better may perform worse in action robustness, challenging existing evaluation paradigms.
Across robotics, World Models (WMs) are increasingly used to evaluate action policies by simulating the consequences of actions in an imagined world, and returning a success or safety verdict. Yet a verdict is only as trustworthy as the WM that produced it, and the WM itself needs to be certified. In video-generation WMs, fidelity metrics such as Fr\'echet Video Distance (FVD) reward visual realism, but ignore whether the world responds correctly to the policy's actions, including those unseen in training. Classical simulation-based validation assumes a trusted simulator evaluating an untrusted policy, whereas generative WMs are themselves unverified learned artifacts. Hence, we argue that any WM used as a test oracle must first be accredited before its verdicts can serve as evidence. Building on credibility practices from safety-critical simulation, including Verification, Validation&Accreditation (VV&A), Safety of the Intended Functionality (SOTIF), and scenario-based testing standards, we define an admissibility ladder (L0-L4) that a WM must climb before its closed-loop verdicts are accepted as assurance evidence. Our framework is embodiment-agnostic, and is instantiated in autonomous driving (AD), where assurance methods for traditional simulation are most mature. Applied to two driving WMs, the lower rungs reveal a reversal: the model that ranks higher on visual generation quality (L0) ranks lower on action-following (L1-L2), so visual fidelity does not predict the action-robustness a closed-loop verdict depends on.
