Search papers, labs, and topics across Lattice.
MagicSim is a unified infrastructure designed for executable embodied interaction, integrating control, skills, and planning into a single deterministic runtime. By utilizing a shared Markov decision process (MDP) and YAML-first specifications, it allows for the construction of diverse executable worlds that facilitate benchmarking, reinforcement learning evaluation, and agent interaction. The key result is a seamless pipeline that transforms commands into grounded trajectories, effectively aligning multimodal representations with executed episodes, thereby enhancing the efficiency of robot learning and interaction tasks.
MagicSim revolutionizes robot learning by merging diverse world construction and execution into a single, efficient framework that enhances both evaluation and interaction capabilities.
Robot learning and embodied agents now require simulation to serve as a shared execution substrate linking control, skills, and planning, not only as a renderer, controller testbed, or fixed task environment. Existing pipelines split these layers with "magic" actions, disconnected training environments, or forward-only renders that cannot reproduce, evaluate, and annotate the same episode. We present MagicSim, an embodied interaction infrastructure built around one deterministic batched runtime and a shared Markov decision process (MDP). From YAML-first specifications that decouple contents, placement, behavior, and agent exposure, MagicSim constructs diverse executable worlds spanning task families, interaction regimes, physics, layouts, sensors, avatars, and robot embodiments in one reset-and-step loop. A common execution interface grounds high-level commands through controllers, atomicskills, planner primitives, and asynchronous planning, realizing them as robot actions rather than simulator-side state edits. One task definition supports three capabilities: benchmark and RL evaluation, an autocollect interface that automatically turns commands into grounded trajectories, and agent/VLM-facing interaction. For automatic execution, commands flow through a Command->Skill->Planner->Robot->Record pipeline, while per-environment command, skill, planning, retry, annotation, and episode states advance independently above the shared physics tick. Successful rollouts are saved as structured multimodal trajectories aligning language supervision, action representations, visual/geometric representations, and task-level status with the executed episode. MagicSim thus unifies diverse world construction, embodied execution, task evaluation, automatic rollout generation, and interactive agent interfaces in one planner-in-the-loop runtime.
