World Models & Planning - Weekly Roundup

Unlock zero-shot sim-to-real transfer for complex legged robots by offloading gait selection to a learned policy that guides a lower-level MPC.

Agentic search gets a meta-RL boost: MR-Search learns to self-reflect and adapt search strategies across episodes, significantly outperforming standard RL baselines.

Multi-robot coverage can now handle multiple sensory demands simultaneously, with provable guarantees on performance even when those demands are initially unknown.

Achieve real-time safety-critical robot control in partially observable environments by decoupling goal reaching, information gathering, and safety into modular, certificate-based components operating directly in belief space.

Stop wrestling with unstable action spaces: ResWM reframes visual RL by predicting incremental action adjustments, leading to smoother control and better performance.

Unlock zero-shot sim-to-real transfer for complex legged robots by offloading gait selection to a learned policy that guides a lower-level MPC.

Agentic search gets a meta-RL boost: MR-Search learns to self-reflect and adapt search strategies across episodes, significantly outperforming standard RL baselines.

Multi-robot coverage can now handle multiple sensory demands simultaneously, with provable guarantees on performance even when those demands are initially unknown.

Achieve real-time safety-critical robot control in partially observable environments by decoupling goal reaching, information gathering, and safety into modular, certificate-based components operating directly in belief space.

Stop wrestling with unstable action spaces: ResWM reframes visual RL by predicting incremental action adjustments, leading to smoother control and better performance.

Forget hand-tuning rollout budgets: $V_{0.5}$ dynamically allocates compute to sparse RL rollouts based on a real-time statistical test of a generalist value model's prior, slashing variance and boosting performance.

Finally, a multi-robot path planning benchmark that lets you directly compare grid-based, roadmap, and continuous planners on the same tasks.

Steer your robot's diffusion policy away from failure modes at inference time with a lightweight performance predictor trained via self-supervised attention.

Forget hand-crafted rewards: this new method learns dexterous manipulation by encouraging the robot hand to explore diverse contact patterns on objects, leading to impressive real-world transfer.

Robust co-design optimization can significantly improve the performance of agile UAVs in real-world environments by directly incorporating uncertainty and disturbances into the design process.

By forecasting compact world dynamics before taking action, DynVLA leapfrogs traditional CoT methods to achieve more informed and physically grounded autonomous driving decisions.

Robots can now learn to manipulate novel objects in dynamic environments by using LLMs to bridge the gap between symbolic planning and reinforcement learning.

Unlock superior trajectories in complex environments with a new ADMM-based solver that jointly optimizes spatial and temporal domains, eliminating the need for complex warm starting.

Incomplete trajectory data got you down? This plug-and-play framework progressively aligns features from incomplete observations with complete ones, boosting prediction accuracy in autonomous driving scenarios.

Achieve 2x better coverage of autonomous driving safety requirements with 6x fewer simulations by automatically generating test scenarios from formal LTLf specifications.

Injecting muscle synergy priors into reinforcement learning drastically improves the realism of simulated human locomotion, even with limited real-world data.

Reaction wheels can dramatically stabilize bipedal hopping robots in low-gravity environments, enabling more consistent upright landings on irregular extraterrestrial terrains.

Achieve significantly higher accuracy and lower mental demand in bimanual teleoperation by intelligently coupling intention estimation with scene-graph task planning and context-aware motion assistance.

Robots can now loosen screws with human-level dexterity thanks to a new framework that combines haptic estimation, online planning, and adaptive stiffness control using a parameterized Equilibrium Manifold.

AI can bridge the gap between simulation and reality in erosion modeling, boosting prediction accuracy by fusing CFD-DEM simulations with experimental data.

By fusing language model reasoning with diffusion-based trajectory generation, KnowDiffuser leapfrogs existing autonomous driving planners on the nuPlan benchmark.

A single meta-RL policy can now handle 66% mass variations and 70% rotor thrust losses in quadrotors, achieving zero-shot sim-to-real transfer for agile maneuvers.

Gaussian trajectory predictors often lie about their confidence, but a new loss function leveraging Kernel Density Estimation can make them more honest, leading to safer autonomous navigation.

By decoupling visual and motor information during pretraining, FutureVLA unlocks more effective visuomotor prediction for vision-language-action models, boosting performance without modifying downstream architectures.

Guaranteeing safety in diffusion-based trajectory planning is now possible by embedding a certifiable barrier function directly into the denoising loop, ensuring forward invariance and preserving the learned path geometry.

By jointly modeling video dynamics and actions, DiT4DiT achieves 10x sample efficiency and 7x faster convergence in robot policy learning, showing that video generation can be a powerful scaling proxy.

Achieve efficient task execution in shared workspaces by interleaving scheduling and motion planning, using symbolic feedback to guide the scheduler towards motion-feasible solutions.

Generate realistic and controllable videos of humans interacting with objects using only sparse motion cues, like wrist positions and object bounding boxes.

Drones can now proactively navigate turbulent environments thanks to a fast wind-prediction framework that integrates geometric perception and local weather data.

Ditch the map: a diffusion model learns to plan UAV swarm trajectories directly from RGB images, enabling reactive and adaptive navigation in cluttered environments.

Forget hand-crafted heuristics: this new dynamics-aware policy learns to exploit contact forces in cluttered environments, outperforming traditional methods by 25% in simulation and showing impressive sim-to-real transfer.

Physics-based dynamics models can make or break sim-to-real reinforcement learning, boosting real-world success by 50% in industrial control tasks where simplified models fail.

For the first time, a famine early warning system offers probabilistic, open-access, continuously running, machine-readable predictions with a commitment to public prospective verification.

Forget hand-engineered reward functions: this method uses language models to learn factorized world states that generalize to new goals and environments, outperforming LLM-as-a-Judge in zero-shot reward prediction.

Offline RL can be made more robust to distribution shift by directly optimizing against worst-case transition dynamics within an uncertainty set, leading to policies that avoid unreliable out-of-distribution actions.

Stop letting simulator errors in critical regions derail your policies: Sim2Act aligns surrogate fidelity with downstream decision impact, leading to more stable and robust decision-making.

By communicating in a shared latent space, Latent-DARM lets you combine the global planning of diffusion models with the fluency of autoregressive models, boosting reasoning accuracy by up to 14% while slashing token usage.

MLLMs still struggle to reliably predict the long-term consequences of actions in egocentric videos, even with structured scene annotations.

LLMs can evolve surprisingly effective, interpretable Python planners that rival state-of-the-art classical planners, at a fraction of the computational cost.

Reconstructing and simulating wind-driven dynamics from video is now possible with a new differentiable framework that enforces fluid dynamics laws.

Robots can now recover from failures during manipulation tasks by explicitly tracking progress against spatial subgoals, without needing extra training data or models.

Self-wrapping cables aren't just a nuisance in robotic manipulation; they're a feature that can be exploited for redirected torque and more efficient object control.

Simulation-based inference can improve neutrino interaction model tuning beyond traditional methods, even suggesting parameter values that better fit experimental data.

By translating visual observations into language, LAP achieves state-of-the-art procedure planning by disambiguating visually similar actions, outperforming vision-only methods.

Humanoid locomotion can be retargeted more realistically by optimizing for dynamics and contact forces, leading to better imitation learning performance.

Latent world models for automated driving are ripe for standardization, and this paper offers a taxonomy and evaluation framework to make them decision-ready.

Text-only foundation models can perform surprisingly well on complex 3D spatial reasoning tasks, rivaling multimodal models, when equipped with a structured spatial representation derived from 3D reconstruction.

RoadLogic automates the creation of diverse, realistic autonomous vehicle test scenarios from declarative specifications, sidestepping the manual effort of imperative approaches.

Autonomous vehicles can now better adapt to the messy, ever-changing real world thanks to a new motion forecasting method that learns new object classes on the fly without forgetting old ones.

Forget costly physical experiments: this framework lets you simulate embodied human-robot interaction to optimize robot designs and controls, unlocking access to internal biomechanical metrics.

Forget pick-and-place: RuleSafe, a new benchmark featuring LLM-generated safe-cracking tasks, exposes the long-horizon planning weaknesses of current robot learning methods.

Ditching VAE bottlenecks for dense DINOv2 features unlocks more stable and accurate visual navigation world models.

Skip the costly robot teleoperation data: ZeroWBC learns surprisingly natural humanoid control policies directly from human egocentric videos.

LLMs can get a 27.8% boost in mathematical reasoning by fusing a hardware-efficient optimal control layer directly into their architecture, enabling planning before prediction.

Quadruped robots can now learn to navigate complex, real-world environments in minutes, not hours, thanks to a new RL framework that prioritizes safety and efficient exploration.

AutoAgent dynamically evolves agent cognition and memory to achieve superior performance in complex, dynamic environments, without requiring external retraining.

Trajectory prediction models can now adapt to new environments far more effectively thanks to a meta-learning approach that dynamically adjusts learning rates based on online data characteristics.

Fuzzy logic can smooth out the sometimes jerky paths generated by A* search, leading to safer and more efficient navigation for unmanned surface vehicles.

By explicitly incorporating stochasticity into physics-informed traffic models, this work provides a more realistic and informative representation of traffic dynamics than traditional deterministic approaches.

Turn your robot's clumsy pre-trained behaviors into expert-level skills with DICE-RL, a surprisingly stable and efficient RL fine-tuning method.

Achieve formally certified collision risk guarantees for robot manipulators in complex, uncertain environments with a novel risk-bounded motion planning framework.

Autonomous driving gets a boost: EvoDriveVLA's collaborative perception-planning distillation framework significantly enhances VLA model performance by tackling perception degradation and planning instability.

A hierarchical OODA loop architecture can significantly improve the adaptability and efficiency of UAV swarms operating in dynamic, uncertain environments.

Autonomous racecars can now overtake rivals 51% faster and with 81% success by predicting their moves and planning dynamically feasible trajectories.

Forget external rewards—this agent learns to explore and adapt by prioritizing its own ignorance, surprise, and staleness, outperforming fixed strategies.

Stop struggling with compounding errors in long-horizon robotic tasks: AtomVLA leverages LLMs and latent world models to decompose tasks and score actions, boosting success rates to 97% on LIBERO.

By explicitly modeling and sharing "execution fidelity" – an estimate of local navigability – VORL-EXPLORE enables multi-robot exploration that avoids bottlenecks and oscillations common in dense, dynamic environments.

Forget expensive, inflexible physical simulators: MRDrive offers an open-source mixed reality platform for in-vehicle HCI research, blending real-world interaction with virtual environments.

Turn your Inspire RH56DFX hand from a black box into a research tool with this characterization, simulation, and control pipeline that achieves 87% grasp success on diverse objects.

LLMs can be used to prune irrelevant information *before* planning, enabling efficient long-horizon multi-robot task planning that outperforms both pure LLM and hybrid LLM-PDDL approaches.

Humanoid robots can now recover from falls with 93% success by baking in classical balance principles into RL, enabling diverse strategies from ankle adjustments to compliant falling.

By disentangling rigid-body mechanics from stochastic interaction effects, STRIDE achieves more accurate and reliable dynamics prediction for robots operating in uncertain environments.

Humanoid robots can now perform complex loco-manipulation tasks with more natural and stable movements by decomposing control into VLM-orchestrated expert policies trained with human motion priors.

GP-PSRL can achieve sublinear regret bounds in continuous control even with unbounded state spaces, resolving prior theoretical limitations and opening the door to more complex RL settings.

By closing the loop with explicit planning and feedback, SPIRAL overcomes the temporal drift and weak semantic grounding plaguing one-shot video generation models.

By intelligently switching between exact and approximate evaluations during genetic programming, HE-GP slashes training time by 17.77% while simultaneously improving the quality of scheduling policies for Earth observation satellites.

The approximation error of spectral RL representations is fundamentally limited by the algebraic connectivity of the state-graph, revealing a crucial topological bottleneck.

Forget explicit labels: this method learns object co-occurrence priors directly from unlabeled visual data, rivaling human search efficiency.

Humanoid robots can now maintain balance under complex external forces without force/torque sensors, thanks to a force-adaptive RL policy that learns to anticipate and compensate for disturbances.

Legged robots can now safely explore unknown, deformable terrain using only proprioceptive feedback to estimate traversability, outperforming traditional methods.

Robots can now learn better world models through unsupervised self-play, outperforming models trained on human data by 40% in failure prediction and 65% in real-world RL.

Ray-tracing simulators can overestimate 5G throughput even with accurate channel predictions, because they fail to capture the real-world adaptation of MIMO spatial layers.

Decoupling reasoning from action generation in autonomous driving VLMs lets you beat larger end-to-end models while slashing training costs.

LLMs can provide quality assurance for reinforcement learning-based search plans in high-stakes missing-child investigations, improving the reliability of AI-driven decision support.

RL agents can completely miss gradual observation drift until it's too late, with a sharp "boiling frog" threshold determining when they finally wake up to the problem.

By verifying high-level symbolic plans with learned continuous dynamics, this neuro-symbolic planner achieves the speed of symbolic methods with the reliability of continuous planning.

Achieve substantially higher success rates in long-horizon mobile manipulation by grounding a vision-language model within a skill-state graph, enabling logically consistent planning and closed-loop replanning.

Forget painstakingly collecting robot data in the real world – this interactive world simulator lets you train policies that perform just as well, but entirely in simulation.

Ditch the comms: This multi-UAV coordination method uses only onboard LiDAR and a perception-aware navigation framework to achieve safe and scalable operation in GNSS-denied environments like dense forests.

By enforcing physical laws, Lagrangian Neural Networks can significantly improve the accuracy and generalization of dynamics models within Model-Based Reinforcement Learning.

By learning to predict blunders from Stockfish evaluations, OGSS enables chess agents to explore more aggressively without sacrificing tactical soundness.

By "imagining" new scenarios and asking "What if this were the true preference?", CRED actively designs environments and trajectories to expose differences between competing reward functions, dramatically improving preference learning.

Ditch the reactive agent: a Llama-2 model fine-tuned to infer semantic zones from object observations enables systematic exploration via TSP optimization, dramatically boosting ObjectNav performance.

By tokenizing trajectories into LLM-friendly point tokens and embeddings, AutoTraces unlocks SOTA long-horizon trajectory forecasting without manual annotation.

Fixed-altitude underwater vehicles can now efficiently search and sample sparse coral colonies thanks to a hierarchical planner that fuses acoustic and visual data.

Achieve a 40% jump in success rates on real-world contact-rich manipulation by intelligently scheduling force feedback into visual-motor policies.

Stripping away seemingly helpful information from agents' observations can actually *improve* the robustness of multi-agent coordination in communication-constrained environments.

Scaling test-time compute can dramatically improve the success rate of robot imitation learning, achieving up to 95% on complex manipulation tasks.

RAMBO's instability got you down? ROMI offers a robust, value-aware model learning approach with implicitly differentiable adaptive weighting that outperforms RAMBO and other SOTA methods in offline RL benchmarks.

A novel self-conditioned GAN learns trajectory forecasting without context, outperforming supervised methods in poorly labeled data by discovering behavioral modes in the discriminator's feature space.