Search papers, labs, and topics across Lattice.
This paper introduces Future-State-Conditioned Vision-Language Navigation (FSC-VLN), which enhances end-to-end vision-language navigation by incorporating future visual information into the training process without requiring access to future images during inference. The authors demonstrate that using a compressed future visual latent as training supervision significantly improves navigation performance metrics such as Success Rate (SR), Optimal Success Rate (OSR), and Success weighted by Path Length (SPL) on the R2R val-unseen dataset, particularly benefiting long-horizon episodes. The findings suggest that aligning a future-query token with visual embeddings can effectively leverage predictive cues for better action selection in navigation tasks.
Future visual cues can dramatically enhance navigation performance, even when not accessible during actual deployment.
End-to-end vision-language navigation (VLN) with causal vision-language models can map instructions and egocentric observations directly to actions, but standard behavior cloning supervises only the next action and does not explicitly train the policy state to be predictive of future visual outcomes. We first ask a diagnostic question: if the policy is given an expert-trajectory future image as privileged input at training and testing time, is that additional visual evidence useful for choosing the current action? (These expert-trajectory future images are unavailable at test time in real deployment, so we use this setting only as a privileged-input diagnostic.) The answer is yes; this sanity check shows that future observations can provide rich, actionable cues. We then ask a deployable question: without accessing future images at inference, can we still benefit from future information by using a compressed future visual latent only as training supervision? We propose Future-State-Conditioned VLN (FSC-VLN), which adds a future-query token and aligns its hidden state to a frozen visual embedding $螖$ steps ahead via a training-only target branch that is removed after training. On R2R val-unseen, FSC-VLN improves SR/OSR/SPL over a StreamVLN-style baseline under two training-data regimes, with larger gains on long-horizon episodes; ablations further support the dual-query design (separating future and action queries).
