Search papers, labs, and topics across Lattice.
This paper details the design and implementation of PGN (Pangu Navigator), a Vision-Language Navigation (VLN) system that utilizes the OpenPangu-7B multimodal foundation model. The authors address the challenges of visual-language alignment and action-space grounding through a two-stage training process, achieving a 62.29% Normalized Action Match (NAM) and a 100% Non-empty Rate (NER) in offline evaluations. These results highlight the effectiveness of their approach in aligning visual and language modalities for navigation tasks, setting the stage for future exploration of error accumulation and path efficiency.
Achieving a 62.29% action match in offline VLN tasks, PGN redefines the benchmarks for multimodal navigation systems.
Vision-Language Navigation (VLN) requires an embodied agent to interpret a natural-language instruction and predict actions from temporally ordered visual observations. Adapting a multimodal large language model to VLN requires visual-language alignment, compact temporal inputs, action-space grounding, and stable training on the target hardware. This technical report presents PGN (Pangu Navigator), an offline VLN action-prediction system built on OpenPangu-7B. Training proceeds in two stages. First, PGMM aligns a frozen EVA-ViT-G/14 vision encoder with the frozen language backbone by training a Q-Former and a two-layer MLP projector. Second, PGN adapts the aligned model to expert navigation trajectories using five-observation windows, epoch-dependent temporal sampling, and a reasoning-then-action output format; this stage freezes the aligned visual pathway and updates three structural-token embeddings and LoRA adapters. The implementation combines mixed-precision computation, selective FP32 computation, and DeepSpeed ZeRO-2 on eight Ascend 910B NPUs. Under teacher-forced, open-loop evaluation on 500 held-out expert trajectories, V9 reports a 62.29% Normalized Action Match (NAM) and a 100.00% Non-empty Rate (NER). These metrics quantify offline expert-action alignment rather than closed-loop navigation success; evaluating error accumulation, path efficiency, and goal completion remains future work.
