Search papers, labs, and topics across Lattice.
This paper introduces TypeGo, an operating-system-style runtime designed for embodied agents that leverages large language models (LLMs) for real-time behavior planning. By structuring LLM-based planning as asynchronous loops that overlap with execution, TypeGo significantly reduces delays in task execution and enhances the management of concurrent goals. The prototype, Kalos, demonstrates a 50% reduction in per-step delay and a 73% decrease in time-to-first-action compared to traditional planning methods, showcasing its efficiency in handling multiple tasks simultaneously.
TypeGo slashes planning delays by over 50% for embodied agents, enabling real-time responsiveness in complex environments.
Large language models (LLMs) can plan behavior for embodied agents from natural language, but treating the LLM as a request/response oracle on the critical path is fundamentally at odds with real-time control and concurrent goals. We argue for an operating-system-style runtime for embodied agents, and instantiate this idea in an early prototype, TypeGo. TypeGo structures LLM-based planning as asynchronous loops at multiple timescales that overlap with execution, and manages the agent's physical body like an OS manages hardware: the Skill Kernel arbitrates typed physical subsystems among concurrent per-task processes, a scheduler preempts them and resumes or replaces each by source, and speculative skill streaming hides LLM latency behind ongoing motion, while a fast first-action path yields visible feedback within a second. Users program behavior through natural language prescriptions that TypeGo dispatches to the LLM-based planners or compiles into low-latency interrupt handlers. Our prototype of Kalos, a Unitree Go2 quadruped, provides preliminary evidence for the design: in our current task suite, it cuts per-step delay by 50% over step-by-step planning and time-to-first-action by 73% over monolithic planning, while admitting concurrent tasks at low scheduling overhead.
