Search papers, labs, and topics across Lattice.
This paper introduces RTNav, an architecture designed for real-time zero-shot object navigation that explicitly incorporates inference latency, asynchronous environment stepping, and bounded compute into its design. The research highlights that existing zero-shot navigation methods experience significant performance degradation when subjected to real-time constraints, emphasizing the inadequacies of traditional synchronous simulator-based approaches. By evaluating RTNav on various real-time navigation tasks, the authors demonstrate improvements in success rates by up to 11% and enhancements in Success weighted by Completion Time by up to 5.1 points compared to prior methods.
Real-time object navigation can see up to an 11% boost in success rates by addressing inference latency and asynchronous stepping in model design.
Navigation in unknown environments to find unforeseen objects has become increasingly feasible with capable vision and language foundation models. However, these models also introduce non-negligible inference latency, which becomes an important concern when agents must operate continuously in the real world. Most state-of-the-art methods are still developed in synchronous simulators, where the environment waits for the agent to act and inference time is effectively free. As a result, agents are often designed around the sequential execution of perception, reasoning, and action, with little regard for time constraints. Under real-time execution, where wall-clock time counts towards the task budget, the inefficiencies of these architectures become clear. We show that recent zero-shot object navigation methods suffer consistent performance degradation under such realistic timing conditions. Motivated by this observation, we propose RTNav, a simple but effective architecture that treats inference latency, asynchronous environment stepping, and bounded compute as explicit design considerations. Evaluated on real-time variants of HM3D-v1, HM3D-v2, and HM3D-OVON, RTNav improves the success rate by up to 11% and the Success weighted by Completion Time by up to 5.1 points over prior work.
