Search papers, labs, and topics across Lattice.
This paper addresses the challenges of applying self-hosted computer-use agents (SHCUAs) for real-time control of unmanned aerial vehicles (UAVs) by restructuring their operation to meet stringent safety and timing requirements. The proposed RT-SHCUA architecture separates semantic reasoning from execution, allowing for mission understanding through slower cloud processing while ensuring that only timely and authorized commands are executed onboard. Evaluation of the prototype demonstrates that this approach maintains task-level responsiveness and enhances safety through mechanisms for trusted admission and evidence preservation.
Transforming SHCUA outputs into contract-bound UAV commands could revolutionize real-time UAV control by ensuring safety and responsiveness in dynamic environments.
Natural-language control offers a promising interface for unmanned aerial vehicles (UAVs), but directly applying self-hosted computer-use agents (SHCUAs) to UAV control introduces a structural mismatch. SHCUAs are designed for interactive host-side tool use, where delayed agent iterations are often acceptable. UAV control, however, is coupled with continuously changing physical states, strict timing constraints, safety risks, and security accountability. A stale, unauthorized, or tampered agent decision may therefore lead to unsafe or untraceable vehicle behavior. This paper proposes a real-time and security-oriented restructuring of SHCUA-based UAV control. Instead of allowing an SHCUA to directly issue flight commands, we transform its outputs into contract-bound UAV skill invocations with explicit timing, state, authority, fallback, and evidence semantics. Based on this abstraction, we design an architecture that separates semantic reasoning from onboard execution and security/safety enforcement. Slow cloud or edge reasoning is used for mission understanding, while onboard components validate and dispatch only timely, authorized, and state-consistent skills. Security-critical enforcement points can be protected by TEE-style or microcontroller isolation mechanisms without moving the full language agent or high-frequency flight-control loop into trusted components. Prototype evaluation shows that RT-SHCUA maintains bounded task-level responsiveness while supporting degraded handling, trusted admission, and auditable evidence preservation for SHCUA-mediated UAV actions.
