Search papers, labs, and topics across Lattice.
This paper introduces IROSA, a framework for adapting robot skills using natural language commands without requiring LLM fine-tuning or direct model-to-robot interaction. IROSA employs a tool-based architecture with an abstraction layer to ensure safety and interpretability while leveraging pre-trained LLMs for tool selection and parameterization. Experiments on a 7-DoF robot performing a bearing ring insertion task demonstrate successful adaptation of speed, trajectory, and obstacle avoidance through natural language.
Control a robot with natural language, no fine-tuning required: IROSA uses a tool-based architecture to safely adapt robot skills using pre-trained LLMs.
Foundation models have demonstrated impressive capabilities across diverse domains, while imitation learning provides principled methods for robot skill adaptation from limited data. Combining these approaches holds significant promise for direct application to robotics, yet this combination has received limited attention, particularly for industrial deployment. We present a novel framework that enables open-vocabulary skill adaptation through a tool-based architecture, maintaining a protective abstraction layer between the language model and robot hardware. Our approach leverages pre-trained LLMs to select and parameterize specific tools for adapting robot skills without requiring fine-tuning or direct model-to-robot interaction. We demonstrate the framework on a 7-DoF torque-controlled robot performing an industrial bearing ring insertion task, showing successful skill adaptation through natural language commands for speed adjustment, trajectory correction, and obstacle avoidance while maintaining safety, transparency, and interpretability.
