Search papers, labs, and topics across Lattice.
LoHo-Manip tackles long-horizon manipulation by using a VLM task manager to decompose tasks into subtask sequences with explicit "done" and "remaining" splits, paired with visual trace prompts (2D keypoint trajectories). This manager operates in a receding-horizon manner, predicting the remaining plan at each step and enabling closed-loop replanning. By conditioning the executor VLA on the rendered trace, the system achieves improved long-horizon success and robustness in both simulation and real-world robotic manipulation.
Forget brittle visual-history buffers: LoHo-Manip uses a VLM task manager with visual trace prompts to achieve robust long-horizon robotic manipulation through implicit closed-loop replanning.
Long-horizon manipulation remains challenging for vision-language-action (VLA) policies: real tasks are multi-step, progress-dependent, and brittle to compounding execution errors. We present LoHo-Manip, a modular framework that scales short-horizon VLA execution to long-horizon instruction following via a dedicated task-management VLM. The manager is decoupled from the executor and is invoked in a receding-horizon manner: given the current observation, it predicts a progress-aware remaining plan that combines (i) a subtask sequence with an explicit done + remaining split as lightweight language memory, and (ii) a visual trace -- a compact 2D keypoint trajectory prompt specifying where to go and what to approach next. The executor VLA is adapted to condition on the rendered trace, thereby turning long-horizon decision-making into repeated local control by following the trace. Crucially, predicting the remaining plan at each step yields an implicit closed loop: failed steps persist in subsequent outputs, and traces update accordingly, enabling automatic continuation and replanning without hand-crafted recovery logic or brittle visual-history buffers. Extensive experiments spanning embodied planning, long-horizon reasoning, trajectory prediction, and end-to-end manipulation in simulation and on a real Franka robot demonstrate strong gains in long-horizon success, robustness, and out-of-distribution generalization. Project page: https://www.liuisabella.com/LoHoManip
