Search papers, labs, and topics across Lattice.
This paper introduces OmniAct, a hierarchical asynchronous architecture designed to enhance the autonomy of embodied agents in unstructured environments by integrating multimodal planning, adaptive memory, and visual preemption. The framework addresses critical limitations in existing systems, such as unbounded context accumulation and lack of failure detection, by providing a unified cyber-physical action space and efficient context management. Experimental results demonstrate that OmniAct significantly improves end-to-end success rates across 40 long-horizon tasks on robotic platforms, while also maintaining efficient token consumption and elevating performance to proprietary levels.
OmniAct achieves unprecedented levels of physical autonomy, outperforming existing systems by seamlessly integrating multimodal planning and adaptive memory management.
Building persistent embodied agents in unstructured environments demands unified orchestration of heterogeneous tools spanning both cyber (APIs, IoT) and physical (manipulation, navigation) domains, coupled with autonomous recovery from physical failures that inevitably arise over extended operation. Existing systems treat these as separate problems: VLM-based planners lack a unified cyber-physical action space, agent frameworks accumulate unbounded context that degrades temporal coherence, and VLA policies execute open-loop without detecting their own failures. We argue that persistent autonomy requires not a monolithic model but a hierarchical asynchronous architecture with explicit separation of planning, memory, and verification. To this end, we present OmniAct, a framework integrating a multimodal semantic planner for skill routing across unified action spaces, an adaptive hierarchical memory with event-boundary-driven compression for sub-linear context growth, and an asynchronous visual preemption engine that closes the semantic loop during physical execution. Across 40 real-world long-horizon tasks on two robotic platforms coordinating four IoT devices, OmniAct achieves consistent improvements in end-to-end success across all complexity levels, maintains near-flat token consumption over under 100k+ accumulated interaction tokens, and elevates mid-scale open-weight models to proprietary-level performance.
