Search papers, labs, and topics across Lattice.
RePlan-Bot, a new Embodied Instruction Following (EIF) agent, uses multi-level replanning to improve performance in long-horizon tasks with irreversible state changes. It integrates a high-level LLM auditor for sub-goal adjustments, a commonsense-guided search using a multi-layered instance map for object localization, and a ViT-based corrector for preemptive action correction. Experiments on the ALFRED benchmark show RePlan-Bot achieves state-of-the-art performance, indicating improved adaptability and reliability in EIF tasks.
LLMs can dynamically replan sub-goals during embodied instruction following, leading to state-of-the-art performance on ALFRED.
Embodied instruction following (EIF) requires agents to understand and execute complex natural language commands within interactive 3D environments. Despite recent advances, existing methods often fail in long-horizon planning and handling irreversible state changes, resulting in low task success rates. To address these challenges, we introduce RePlan-Bot, a novel EIF agent that performs multi-level, continuous replanning throughout task execution. RePlan-Bot integrates a high-level LLM-based auditor for dynamic sub-goal adjustments guided by environmental feedback, a commonsense-guided search mechanism based on a multi-layered instance map for precise and structured object localization, and a lightweight ViT-based corrector to preemptively fix risky low-level actions. Evaluated on the ALFRED benchmark, RePlan-Bot achieves state-of-the-art performance in both seen and unseen environments, demonstrating superior adaptability and reliability.
Tsinghua AI