Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces RoboTALES, a novel framework that enhances robot policy learning by generating task-aligned simulated futures through a combination of hierarchical LLM-based planning and VLM-based evaluation. By breaking down complex tasks into manageable subgoals and providing reward-based feedback, RoboTALES ensures that the imagined futures remain focused on the task intent, leading to improved action coherence. Evaluations on manipulation tasks demonstrate that RoboTALES significantly outperforms existing methods, particularly in long-horizon scenarios, showcasing its effectiveness in visuomotor control.
Task-aligned simulated futures can dramatically improve robot policy learning, yielding superior performance in complex manipulation tasks.
Pretrained video generative models are promising backbones for visuomotor control, but their imagined futures often drift from task intent and are not reliably action-conditional. As a result, these models can be difficult to use for planning or policy extraction. To address these limitations, we propose RoboTALES, a single-stage framework that learns task-aligned simulated futures and uses them to train robot policies. Our approach introduces two key innovations: (1) a hierarchical LLM-based planner that breaks complex tasks into a sequence of subgoals to guide the model's imagination; and (2) a VLM-based critic that evaluates these ``imagined''futures and uses reward-based feedback to keep the model's internal representations focused on the goal. By anchoring the video generator in abstract reasoning, we produce temporally consistent rollouts and more coherent actions. We evaluate RoboTALES on diverse manipulation tasks from RoboCasa and LIBERO10, and show that our method consistently outperforms existing methods, especially in long-horizon tasks. Our code and models are publicly available at https://github.com/hananshafi/RoboTALES.