Search papers, labs, and topics across Lattice.
This paper introduces STEP, a State-aware Task Estimator and Planner that enhances human-robot collaboration by enabling Multi-modal Large Language Models (MM-LLMs) to explicitly estimate system states and predict state transitions during task planning. By addressing the limitations of MM-LLMs, which often lead to hallucinated actions and ambiguous plans, STEP generates more accurate and actionable long-horizon plans. In evaluations on a simulated robot assembly task, STEP outperformed the state-of-the-art by 32.8% in action executability and 14.8% in final-state error, demonstrating its effectiveness in practical applications.
Explicitly incorporating state awareness into task planning with MM-LLMs leads to a 32.8% increase in action executability, revolutionizing human-robot collaboration.
Effective human-robot collaboration in industrial settings requires robots to understand human intentions and assist with task planning, reducing workload. Recent works have explored the use of Multi-modal Large Language Models (MM-LLMs) for task planning in such data-scarce scenarios, leveraging in-context learning to interpret user actions and generate long-horizon action plans in natural language. However, MM-LLMs inherently lack an understanding of system states and do not track state transitions, often leading to hallucinated actions that deviate from the intended goal. Additionally, generating action plans in natural language tends to limit the generated plans to a high level, introducing ambiguity in action execution. To address these limitations, we propose the State-aware Task Estimator and Planner (STEP), which prompts a MM-LLM to explicitly estimate the state of the system and predict the state transitions resulting from executed actions. By forecasting future states alongside actions, STEP ensures task-convergent planning while also providing additional assistance parameters necessary for executing the predicted actions. We evaluate STEP in a simulated environment using a robot assembly task. Our approach outperforms the state-of-the-art by 32.8% in action executability and 14.8% in final-state error.
