Search papers, labs, and topics across Lattice.
MistyPilot, an agentic LLM framework, is introduced to enable social robots to autonomously select, orchestrate, and configure tools based on high-level user instructions. The framework employs a Physically Interactive Agent (PIA) for task execution and a Socially Intelligent Agent (SIA) for dialogue, integrating a fast-slow thinking paradigm to optimize user preference capture, latency, and task efficiency. Experiments across five benchmark datasets demonstrate MistyPilot's effectiveness in routing, task completion, retrieval efficiency, tool scalability, and emotion alignment.
Social robots can now autonomously orchestrate complex tasks with improved efficiency and emotional alignment, thanks to a novel fast-slow thinking LLM framework.
With the availability of open APIs in social robots, it has become easier to customize general-purpose tools to meet users' needs. However, interpreting high-level user instructions, selecting and configuring appropriate tools, and executing them reliably remain challenging for users without programming experience. To address these challenges, we introduce MistyPilot, an agentic LLM-driven framework for autonomous tool selection, orchestration, and parameter configuration. MistyPilot comprises two core components: a Physically Interactive Agent (PIA) and a Socially Intelligent Agent (SIA). The PIA enables robust sensor-triggered and tool-driven task execution, while the SIA generates socially intelligent and emotionally aligned dialogue. MistyPilot further integrates a fast-slow thinking paradigm to capture user preferences, reduce latency, and improve task efficiency. To comprehensively evaluate MistyPilot, we contribute five benchmark datasets. Extensive experiments demonstrate the effectiveness of our framework in routing correctness, task completeness, fast-slow thinking retrieval efficiency, tool scalability,and emotion alignment. All code, datasets, and experimental videos will be made publicly available on the project webpage.
