Search papers, labs, and topics across Lattice.
This paper introduces the Semantically Rich World Model (SR-WM), which enhances physical interaction planning by structuring world models around five functional roles: gripper, target, goal, relation, and phase. By utilizing a visual entity encoder to extract soft entity hypotheses and a role binder to assign task-specific roles, SR-WM enables a more nuanced understanding of entity interactions beyond mere identification. The model's ability to predict role transitions and fine-grained semantics significantly improves multi-candidate action generation and planning accuracy across various simulation environments.
Transforming object-centric prediction into a semantic interface, SR-WM enables planning-oriented decision-making that captures the complex roles of entities in physical interactions.
World models for physical interaction are typically trained to predict future observations or latent features; however, a planning-oriented model must answer a fundamentally different question: whether a candidate action produces a task-consistent future while preserving essential relations.Monolithic state representations obscure the underlying entities, while standard instance-level object slots merely identify \emph{what} is present without specifying \emph{what role} each entity plays in the task context. To bridge this gap, we present the Semantically Rich World Model (SR-WM), a task-conditioned world model structured around five functional roles: gripper, target, goal, relation, and phase.Within SR-WM, a visual entity encoder extracts soft entity hypotheses from pretrained patch features, allowing segmentation masks to serve as optional proposal priors without mandating them as required state representations or inference inputs.A role binder subsequently maps these hypotheses to task-specific roles, while an action-conditioned dynamics model predicts role transitions alongside fine-grained semantics, including grasp/contact, predicate establishment, relation preservation, fixture state, and phase change.Crucially, this unified role state grounds downstream multi-candidate action generation, stage-aware reranking, and violation-aware suffix resampling.Our comprehensive evaluation protocol spans all four LIBERO simulation suites, cross-suite transfer, perception diagnostics, and action-sensitivity analysis.Ultimately, this formulation transforms object-centric prediction into a semantic interface linking visual dynamics with planning-oriented decision making.
