Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To bridge the gap between compute-heavy vision-language navigation (VLN) models and resource-constrained embodied hardware, the authors develop EdgeVLN, combining an aggressively quantized StreamVLN backbone with an auxiliary trajectory termination head executed via an optimized edge inference engine. System deployment on edge platforms has historically been hindered because simple model compression fails to address runtime memory overhead and execution-path energy spikes. Evaluated on an NVIDIA Jetson Orin NX across R2R-CE val-unseen, their 4-bit implementation (IQ4 NL) achieves a 58.02% success rate—surpassing the unquantized BF16 baseline—while fitting into 11.35 GB of RAM, accelerating inference 20.8×, and slashing step energy 13.3×.
Quantized 4-bit vision-language navigation not only fits onto a 16 GB edge device, it actually outperforms the unquantized BF16 baseline while cutting energy by 13.3× and running over 20× faster.
Vision-language navigation (VLN) models perform well but target compute-rich platforms, limiting deployment on memory- and power-constrained robotic edge devices. Compression alone does not establish whether a VLN model fits the memory, latency, and energy budgets of an edge platform while preserving navigation behavior. We introduce EdgeVLN, a runtime-aware, deployment-ready quantized VLN model that closes this gap. EdgeVLN combines a quantized StreamVLN model with Latent Trajectory Termination Extractor (LATTE), a lightweight causal transformer that improves real-time stopping by predicting a Stop Action verifier rank. Both execute through our llama.cpp VLN driver, which reconstructs streaming context and prunes memory tokens on-board. We characterize a pretrained StreamVLN backbone across weight quantization from 8 to 2 bits and multiple inference runtimes to identify a feasible operating point. LATTE reuses backbone hidden states within the budget freed by quantization, requiring neither a second vision encoder nor an additional backbone forward pass. We evaluate six backbone precisions and seven candidate stop heads on BF16 and IQ4 NL across all 1,839 R2R VLN-CE val-unseen episodes. We measure success rate (SR) in simulation and latency, energy, and resident memory on an NVIDIA Jetson Orin NX 16 GB. LATTE achieves our highest SR, 58.02 percent on the deployed 4-bit model, exceeding the BF16 baseline with only 0.013 s additional latency per navigation step. Four-bit formats achieve nearly identical SR, but step energy varies 36.8 times by execution path. Only IQ4 NL under our VLN driver fits the board, using 11.35 GB resident memory while running 20.8 times faster and using 13.3 times less energy than storage-streamed BF16. INT2 collapses. Runtime selection, memory-token pruning, and quantization are essential for efficient edge deployment.