World Models & Planning - Weekly Roundup

Latent world models scale predictably with compute via a second-order power law, enabling accurate forecasting of robotic planning performance and zero-shot task execution on real hardware.

The hardest-to-distinguish competitor, not the average, dictates the speed of decentralized adaptive sensing.

Autoregressive video pretraining unlocks the full potential of long contexts for real-time robot control, boosting success rates by 15% and enabling complex manipulation previously impossible.

A single neural network now predicts energy use for millions of diverse buildings with physics-validity and accuracy rivaling black-box models, all while being orders of magnitude smaller.

Average-reward RL for general multichain MDPs is now possible without model knowledge or reduction to discounted problems.

Latent world models scale predictably with compute via a second-order power law, enabling accurate forecasting of robotic planning performance and zero-shot task execution on real hardware.

The hardest-to-distinguish competitor, not the average, dictates the speed of decentralized adaptive sensing.

Autoregressive video pretraining unlocks the full potential of long contexts for real-time robot control, boosting success rates by 15% and enabling complex manipulation previously impossible.

A single neural network now predicts energy use for millions of diverse buildings with physics-validity and accuracy rivaling black-box models, all while being orders of magnitude smaller.

Average-reward RL for general multichain MDPs is now possible without model knowledge or reduction to discounted problems.

Generative AI now predicts complex 3D urban wind and temperature fields with LES-level accuracy in seconds, unlocking rapid, turbulence-aware urban design.

A scalar adjoint method slashes the computational cost of fine-tuning flow policies in RL, achieving state-of-the-art results on complex manipulation tasks.

Even perfect AI agents fail to guarantee safety in realistic, time-sensitive, and partially observable environments, fundamentally challenging agent-agnostic alignment approaches.

Logarithmic regret is achieved in adaptive control of unknown, changing systems by exploiting the unique energy signatures of admissible plant changes during exploitation.

The optimal strategy for the cow-path problem is provably cyclic, meaning no search algorithm can outperform a fixed path traversal order.

A frozen graph backbone effectively preserves past structural knowledge in continual multi-agent reinforcement learning, dramatically reducing forgetting and boosting transfer across tasks with distinct graph structures.

Quadrupedal robots can achieve up to 20% lower energy consumption by learning speed-dependent gaits with a hierarchical RL policy that explicitly optimizes cost of transport.

Planning in visual world models can be accelerated by $5\times$ by pruning $50\%$ of spatial tokens based on their influence on the planning cost gradient, without sacrificing performance on most control tasks.

Co-evolved communication protocols exhibit surprising robustness when transferred from 2D to 3D physics, with performance scaling linearly with episode length and social cues acting predominantly as jam-assistance.

Interactive video generation can be sped up by 1.4x by intelligently caching transformer residuals based on detected control transitions, without sacrificing quality.

A simulation flywheel recursively bootstraps embodied data, generating richer scenes and tasks for robot pretraining and evaluation.

Most video world models fail basic physics tests, with even the best models exhibiting significant physical inconsistencies across mechanics, optics, and fluids.

Agents can learn to act reliably in new environments by generating their own practice tasks and distilling reusable heuristics from self-identified failures, achieving significant performance gains without human supervision or pre-defined benchmarks.

Embodied agents can now reliably achieve complex, multi-stage goals in dynamic environments by learning to predict and ground targets from decoupled visual context, even when the goal is out of view.

Spatiotemporal time series forecasting now boasts valid confidence intervals even with imputed labels and temporal dependencies, overcoming a critical limitation of existing PPI.

Sequential hypothesis testing without explicit densities is now possible with anytime-valid error control and near-optimal performance.

Occupancy measure formulation enables decision-focused learning in MDPs with closed-form gradients, overcoming KKT-based scalability bottlenecks and achieving state-of-the-art regret.

Unlocking broad downstream generalization for 3D atomistic systems, Atom-JEPA achieves state-of-the-art property prediction by learning from unlabeled structures alone.

Interactive AI worlds can now speak and react in real-time with spatially accurate sound, bridging the sensory gap in generative environments.

Segment reward feedback can drastically reduce RL regret under binary feedback by increasing segment granularity, a benefit surprisingly absent with sum feedback.

LLM-generated heuristics can now guarantee optimal plans, outperforming specialized planners by combining generative power with formal verification.

End-to-end driving models can learn interpretable safety costs over reachable futures, drastically reducing collisions without fine-tuning.

LLMs can decode partner intents but struggle to translate this knowledge into cooperative actions, with explicit action recommendations proving far more effective than abstract rule statements.

Robot world models can now predict photorealistic RGB *and* accurate metric depth, bridging the gap between visually plausible rollouts and geometrically consistent 3D environments.

Web agents trained with adaptive adversarial simulation achieve 33.6% higher completion rates against unseen prompt injection attacks, even outperforming frontier adversaries.

Language instructions now directly control real-time, high-fidelity 360° video generation, enabling fluid interactive exploration from a single panorama.

External context is often unnecessary for model-based RL agents that implicitly learn to recover latent environment dynamics from their own experience.

Semantic decision engines routinely fail network deadlines and select infeasible actions, with most claims of real-time performance unsupported by evidence.

Geometry-guided retrieval unlocks persistent memory in linear-attention video world models, boosting long-horizon consistency by 0.74 dB PSNR and 28.5% rotation error reduction without retraining.

Achieve 25x speedup in World Action Models with <1% performance drop by enabling one-step action generation and asynchronous expert execution.

Hierarchical JEPAs unlock efficient long-horizon visual planning by learning distinct latent spaces for different temporal scales, dramatically boosting success rates and reducing compute.

Learned AI simulators can pass standard forecast metrics yet fundamentally fail to capture system dynamics, as demonstrated by a new response theory probe that reveals critical divergences in forced responses.

Deterministic weather foundation models can be efficiently transformed into powerful probabilistic forecasters, generating skillful 15-day forecasts in minutes.

Asynchronous execution fundamentally alters action semantics for world-model controllers, causing planning models to hallucinate future actions and recurrent models to misattribute transitions.

Explicitly modeling the temporal rate of change in spatio-temporal graph models unlocks superior prediction accuracy and pattern preservation for propagating phenomena.

TSFMs for predictive control fail to capture system dynamics without explicit control excitation in their inference context, even with low forecasting error.

Robots can learn to sacrifice immediate gains for superior team coordination, outperforming centralized and reactive planners even with over a thousand agents.

A single, static adversarial patch can cripple world action models across diverse tasks and architectures by inducing a stable, global representation shift, even without querying the target policy.

LLMs' apparent behavioral fidelity in sequential tasks can mask fundamentally incorrect generative mechanisms, failing to recover latent structure even with extended context.

Directly learning robot action policies from raw video without explicit action labels unlocks scalable world action models.

Autoregressive video models can now perform explicit, goal-directed reasoning by jointly predicting the final outcome and intermediate steps, achieving state-of-the-art performance with 75% fewer training steps.

A single set of learned manipulation skills transfers to the real world with 78% success, bypassing the need for task-specific fine-tuning or re-training on real data.

Current omnimodal video embeddings struggle with physical reasoning, yet specific physics can be recovered and leveraged to significantly improve the physical fidelity of generated videos.

Robot skills can be dramatically improved post-training by autonomously generating practice tasks, diagnosing failures with privileged simulator states, and iteratively refining symbolic skills and system prompts, achieving near-perfect success on novel manipulation tasks.

Persistent world models can now track objects even when they leave the agent's view, thanks to a novel actor-observer decoupling mechanism.

Achieve precise, intuitive 3D control over video generation by moving rigid object meshes, eliminating depth ambiguity and viewpoint inconsistencies.

Humanoid robots can learn entirely new complex manipulation skills at test-time by evolving reward programs, unlocking latent capabilities of pre-trained controllers.

Robust optimal policies and values for RMDPs are now computable in strongly polynomial time via a single LP, unifying and extending prior results across diverse uncertainty models.

Abstract future prediction in latent space significantly boosts LLMs' chemical reasoning and molecular outcome prediction capabilities.

Judgment models can solve complex problems when the answer is directly evaluable from the input, but fail catastrophically when simulation is required, even if they possess the underlying knowledge.

Discover the "why" behind ML benchmark gains with MECHVAR, a variance-guided approach that achieves near-optimal mechanism discrimination 100x faster than EIG.

Model-free Q-learning now guarantees optimal reachability policies in a broad class of MDPs, slashing sample complexity and memory needs.

Current egocentric video generation models fail to execute multi-step manipulation tasks realistically, often skipping crucial actions and exhibiting poor physical dynamics.

End-to-end learning of temporally predictable latent spaces for world models unlocks superior future scene understanding without the need for separate representation and prediction training stages.

A novel policy optimization method bypasses critic action gradients entirely, achieving faster, more robust updates and improved exploration in continuous control.

A robot can learn its partner's hidden physical limitations simply by watching it collaborate, enabling seamless coordination on novel tasks without explicit communication.

Intrinsic motivation can be learned from system dynamics alone, unlocking unsupervised discovery of essential robotic behaviors and complex locomotion.

Jointly training on simulation and experimental data unlocks balanced prediction in physical systems, outperforming sequential transfer and preserving crucial simulation-only knowledge.

Diffusion models can plan collision-free, multi-agent trajectories without any training data by leveraging analytical local scores derived from geometric constraints.

Ghost nodes and physics refiners unlock stable, accurate long-horizon hydrologic forecasts even with missing upstream boundary data.

Learned QKD attacks exploiting device drift can extract nearly 99% of the theoretical maximum information, shattering assumptions based on stationary channels.

Standard intrinsic rewards can fail to drive optimal exploration, even in simple settings, by neglecting the value of counterfactual information.

Robots can now plan complex, long-horizon tasks in novel environments by predicting sparse visual sub-goals, achieving 70% success in zero-shot real-world experiments.

An understanding-driven approach unlocks systematic spatial-memory management for world models, dramatically improving long-video generation quality.

Rejection-driven resubmission, not just population growth, is a primary driver of overwhelming reviewer burden in academic research.

Unified LLMs can now achieve state-of-the-art 3D generation and understanding by treating geometry as an explicit sequence of sparse octree tokens, without compromising core language abilities.

A single capable verifier can dramatically improve terminal agent action reliability, achieving higher success rates at lower costs than simply generating more trajectories.

Autonomous driving diffusion planners now learn to intrinsically adhere to driving rules and explain their decisions, significantly improving safety in challenging scenarios.

Learned world models can achieve real-time control in live games, outperforming behavioral cloning and maintaining high-frequency action loops.

Howard's policy iteration for deterministic MDPs is not strongly polynomial, exhibiting an exponential iteration complexity that separates it from the simplex method.

Sonar-only SLAM now achieves robust topological mapping by verifying loop closures with a sequence-aware approach, overcoming the inherent ambiguity of acoustic echoes.

Radar novel-view synthesis finally unlocks Doppler, achieving 90.7% object detection recovery by separating scene geometry from sensor spread and enabling zero-shot configuration transfer.

Agents can now reliably navigate evolving environments by maintaining a probabilistic 4D belief that accounts for hidden target movements and revises predictions based on intermittent, visibility-aware observations.

Robots can now learn from physical failures in real-time, autonomously revising their own capabilities to achieve unprecedented long-horizon manipulation success.

An optimal threshold policy for irrevocable matching in growing trees loses at most twice the total-variation error of the true growth law, even when forecasts are misspecified.

Humans can directly control complex embodied agents using a handful of learned, composable skills discovered through self-play, bypassing the need for low-level action control.

Imagined failures in a world model can pinpoint exactly which robot skill components need new demonstrations, boosting success from 13% to 75% with minimal data.

Language, when self-evolving and treated as a physical representation, can unlock state-of-the-art physical plausibility in video world models, even surpassing proprietary systems.

Separating policy roles in offline RL unlocks significant performance gains by tailoring updates for value bootstrapping versus direct action execution.

LLM agents trained solely in rule-generated fictional worlds achieve state-of-the-art transfer to real-world search tasks, outperforming agents trained on actual data.

Tree search on on-policy trajectories dramatically boosts policy gradient learning by uncovering rare high-return paths, outperforming PPO on Atari and MuJoCo with minimal bias increase.

Neural predictors can be made dynamically feasible without ground-truth controls by a frozen, post-hoc operator that enforces learned dynamics manifolds via iterative control correction.

Hybrid RL for bus holding suffers when simulators misrepresent passenger journey completion, leading to worse outcomes than purely offline methods.

State representation quality, not goal representation, is the critical bottleneck in offline navigation GCRL, with simple positional encodings boosting performance significantly.

Autonomous agents can recursively self-improve LLM-generated games, achieving superhuman performance and drastically cutting generation costs by internalizing development experience.

A single video tokenizer can achieve superior semantic alignment and fidelity compared to much larger models by directly conditioning on frozen visual features.

Streaming world models can now maintain consistent representations across minutes-long revisits without quadratic attention costs by leveraging a novel retrieval-based memory bank.

Reconstructing revisited scenes in streaming world models is now possible with constant memory, outperforming full-softmax attention by 30% on peak memory.

A single visual interface, action views, bridges the gap between video generation models and diverse robotic embodiments, enabling unified action modeling and execution.

Most current robotic manipulation policies fail to retain crucial historical information, leading to significant performance drops in partially observable tasks, but a novel memory framework boosts success rates across diverse challenges.

A 4B grounding foundation model achieves state-of-the-art visual grounding, outperforming larger models and boosting downstream robotic and autonomous driving performance by up to 24.8% in OOD settings.