World Models & Planning - Weekly Roundup

Forget generating plausible-but-fake details: 3DreamBooth bakes a robust 3D prior into video generation models using only a single-frame optimization, enabling truly view-consistent customized subject videos.

Adding the T-pentomino to Tetris Block Puzzle makes the game significantly harder, quantified by a slowdown in SGAZ agent convergence.

Maximizing entropy of future state-action visitations boosts feature coverage within single RL trajectories, offering a new exploration strategy.

Coordinating multi-robot teams to complete manipulation tasks just got easier: GoC-MPC handles dynamic task assignments and disturbances without training data or environment models.

VAE-GANs let you have your cake and eat it too: high-fidelity geological models *and* accurate history matching in reservoir simulation, something previous DL methods couldn't deliver.

Forget generating plausible-but-fake details: 3DreamBooth bakes a robust 3D prior into video generation models using only a single-frame optimization, enabling truly view-consistent customized subject videos.

Adding the T-pentomino to Tetris Block Puzzle makes the game significantly harder, quantified by a slowdown in SGAZ agent convergence.

Maximizing entropy of future state-action visitations boosts feature coverage within single RL trajectories, offering a new exploration strategy.

Coordinating multi-robot teams to complete manipulation tasks just got easier: GoC-MPC handles dynamic task assignments and disturbances without training data or environment models.

VAE-GANs let you have your cake and eat it too: high-fidelity geological models *and* accurate history matching in reservoir simulation, something previous DL methods couldn't deliver.

Predictive policing algorithms can exhibit extreme racial bias, with one city showing a 157x higher detection rate for one racial group in a single year.

Discovering an agent's hidden intentions is now possible by analyzing their interventions within a causal model, revealing the "why" behind their actions.

Gaussian assumptions about Earth structure introduce bias and significantly under-report moment tensor uncertainties, but simulation-based inference offers a robust alternative for more reliable earthquake source characterization.

Forget hand-crafted assets and heuristics: V-Dreamer uses video generation models to automatically create diverse, physically plausible robotic simulation environments and trajectories directly from language.

Differentiable collision checking in configuration space, previously a major hurdle, is now achievable with zero-shot generalization thanks to CSSDF-Net.

Achieve more physically realistic video generation by explicitly modeling 3D geometry and physical attributes across multiple viewpoints.

Humanoid robots can now traverse complex terrains with human-like gaits, thanks to a surprisingly simple and efficient framework that eschews adversarial training.

Achieve 9x lower trajectory error and 3x better FID in motion generation by using a diffusion-based discrete motion tokenizer that elegantly handles both semantic and kinematic constraints.

Autonomous driving models can be made significantly more robust and safe by explicitly de-confounding their training via causal intervention, eliminating reliance on spurious correlations.

Stop leaking your secrets to the cloud: PlanTwin lets LLM agents plan over your private data without actually exposing it.

Ditch the heavyweight controllers: these lightweight MPC approaches bring real-time attitude synchronization to resource-constrained spacecraft.

Differentiable environments and backpropagation offer a surprisingly effective alternative to reinforcement learning for AAV trajectory optimization, sidestepping credit assignment problems.

Decentralized MPC with control barrier functions lets multi-robot quadrupeds safely navigate complex environments in real-time, achieving performance on par with centralized approaches but with significantly reduced computation.

Digital twins can now discriminate between different types of cyberattacks on critical infrastructure, enabling targeted responses instead of costly full shutdowns.

LLMs can navigate more efficiently in unfamiliar environments by reasoning over a tree of possible paths, not just isolated waypoints, enabling them to consider en-route information gain and prune unpromising branches.

Robots can learn faster and generalize better by encoding dynamics directly into their neural network architecture, outperforming standard transformers and GNNs.

Forget painstakingly designing simulation environments: generative 3D world models let you RL-fine-tune robot VLAs with massive scene diversity, boosting real-world transfer by 3x.

Unlock real-time control for massive multi-agent swarms: this method slashes computation from cubic to linear with horizon length, making long-horizon density-driven control practical.

Neural solvers can now effectively handle the complexities of multi-agent coordination and multi-objective trade-offs in routing problems, outperforming traditional heuristics.

MLLMs can gain surprisingly strong 3D spatial reasoning abilities simply by tapping into the latent knowledge already present in video generation models.

Optimal multi-agent path planning with asynchronous actions is now provably complete, sidestepping the theoretical incompleteness of prior continuous-time approaches.

Guaranteeing safety in spacecraft autonomy is now more tractable: a CBF-CLF informed imitation learning approach achieves NMPC-level performance with real-time feasibility on commodity hardware.

Agents can now "hallucinate" optimal viewpoints for reasoning by storing and re-rendering scenes with 3D Gaussian Splatting, enabling recovery from initial observation failures.

Hierarchical memory, inspired by human cognition, beats standard approaches in robotic manipulation tasks requiring both precise tracking and long-term retention.

Robots can now manipulate objects with greater dexterity and adaptability thanks to a new world model that leverages both vision and high-frequency tactile feedback to predict and react to contact dynamics.

Standard DRL collapses in volatile environments because it mistakes irreducible noise for a lack of data, but RE-SAC fixes this by explicitly separating these uncertainties.

Robots can now train in realistic, thermally-accurate simulated fires, paving the way for safer and more reliable real-world firefighting deployments.

Achieve real-time online learning for model predictive control with a novel spatio-temporal Gaussian Process approximation that maintains constant computational complexity.

By iteratively reasoning over video snippets with a Chain-of-Thought, $\text{R}^2$VLM achieves state-of-the-art long-horizon task progress estimation without needing to process entire videos at once.

Ditching rigid digital twins for adaptable world models could unlock truly intelligent edge computing in 6G networks.

By treating 3D scene editing as goal-regressive planning rather than pure generation, Edit-As-Act achieves instruction fidelity, semantic consistency, and physical plausibility that existing methods miss.

Legged robots can navigate more reliably with noisy sensors thanks to a new state estimator that avoids Gaussian noise assumptions.

Achieve stable, real-time kilometer-scale autonomous driving simulations by generating vector-graph tiles incrementally using a novel diffusion flow approach.

Seemingly accurate physics-informed surrogates can fail spectacularly when integrated into power system simulations, especially under stress, highlighting the need for rigorous in-simulator validation.

Generate consistent stereo videos directly from RGB data, bypassing depth estimation and monocular-to-stereo conversion, with StereoWorld's novel camera-aware attention mechanisms.

Representing highly nonlinear vehicle dynamics in a lifted linear space via Koopman operator theory enables state-of-the-art long-term state estimation for complex electric trucks.

Simulate earthquake ground motion 10,000x faster with a new latent operator flow matching method, opening the door to real-time risk assessment for critical infrastructure.

Forget rigid physics engines, this badminton RL environment uses real player data to simulate realistic rallies and strategic gameplay.

Heuristic maritime routes lead to extreme fuel waste in nearly 5% of voyages, but this RL approach cuts that risk by almost 10x.

LLMs in embodied environments get a massive boost from structured rules, with rule retrieval alone contributing +14.9 pp to single-trial success.

VLN agents can navigate more effectively by predicting their future states and proactively planning based on forecasted semantic map cues, rather than relying solely on historical context.

Encoding deformable object dynamics with particle positions unlocks sim-to-real transfer for manipulation tasks, achieving impressive real-world success rates.

Drones can now land safely in complex, unknown environments using only a camera, thanks to a new system that dynamically maps and reacts to surroundings in real-time.

Ditch fixed compute budgets: this new flow-matching method for robotic control adaptively allocates computation, speeding up simple tasks and focusing on complex ones.

ManiDreams lets robots handle real-world uncertainty in manipulation tasks without retraining, outperforming standard RL baselines under various perturbations.

Robot world models can be significantly improved by directly rewarding them for generating videos that lead to physically plausible robot actions, even if the videos themselves contain visual artifacts.

A complete autonomy stack enables centimeter-level localization and mapping on the moon, even without GPS.

Finally, a rigorous RL benchmark: generate environments with *provably* optimal policies, enabling controlled algorithm evaluation against ground truth.

Accurately predict urban pollutant dispersion in real-time with a novel data-driven model that's orders of magnitude faster than traditional CFD.

Demonstrator diversity unlocks the ability to learn latent actions and dynamics from offline RL data, even without explicit action labels.

LLMs can be economically aligned to real-world consumer preferences via post-training on transaction data, enabling more accurate and stable economic simulations.

By cleverly turning novel view synthesis into a self-supervised inpainting problem, VisionNVS eliminates the need for ground truth images of novel views, outperforming LiDAR-dependent baselines.

Forget finetuning: DynaEdit unlocks complex video edits like action modification and object insertion, all without training, using clever manipulation of pretrained text-to-video models.

Achieve zero-shot adaptation to new tasks in complex control environments by learning a shared low-dimensional goal embedding that unifies policy and value function representations.

NeRFs can now guide extraterrestrial rovers around unexpected obstacles, thanks to a novel planning framework that blends local observations with global terrain understanding.

Q-value policies, traditionally outperformed by state-value policies in planning, can surpass them with the right regularization, offering a faster alternative for policy evaluation.

Robots can now plan 9x faster and achieve significantly higher success rates by decoupling action prediction from video generation in World-Action Models.

A new mixed reality testbed lets you plug real human drivers into a CAV simulation, offering unprecedented realism for testing autonomous vehicle interactions.

Guaranteeing robot safety and task completion just got easier: this method enforces complex temporal logic constraints on pre-trained robotics models without any fine-tuning.

Human unpredictability is now a feature, not a bug: a mixed-reality testing framework leverages human interaction to generate high-quality corner cases for vehicle-infrastructure cooperation systems.

Autoregressive neural surrogates can now simulate dynamical systems for infinitely long horizons, thanks to a novel self-refining diffusion model that avoids error compounding.

Ditch the data augmentation and decoders: R2-Dreamer's Barlow Twins-inspired objective delivers faster, more versatile MBRL, especially when spotting the small stuff matters.