CASIA OpenIR  > 模式识别实验室
Neighbor-view Enhanced Model for Vision and Language Navigation
Dong An; Yuankai Qi; Yan Huang; Qi Wu; Liang Wang; Tieniu Tan
Conference NameACM International Conference on Multimedia
Source PublicationProceedings of the ACM International Conference on Multimedia
Conference Date2021-10-20
Conference PlaceChengdu, China

Vision and Language Navigation (VLN) requires an agent to navigate to a target location by following natural language instructions. Most of existing works represent a navigation candidate by the feature of the corresponding single view where the candidate lies in. However, an instruction may mention landmarks out of the single view as references, which might lead to failures of textualvisual matching of existing methods. In this work, we propose a multi-module Neighbor-View Enhanced Model (NvEM) to adaptively incorporate visual contexts from neighbor views for better textualvisual matching. Specifically, our NvEM utilizes a subject module and a reference module to collect contexts from neighbor views. The subject module fuses neighbor views at a global level, and the reference module fuses neighbor objects at a local level. Subjects and references are adaptively determined via attention mechanisms. Our model also includes an action module to utilize the strong orientation guidance (e.g., “turn left”) in instructions. Each module predicts navigation action separately and their weighted sum is used for predicting the final action. Extensive experimental results demonstrate the effectiveness of the proposed method on the R2R and R4R benchmarks against several state-of-the-art navigators, and NvEM even beats some pre-training ones. Our code is available at

Indexed ByEI
IS Representative Paper
Sub direction classification机器人感知与决策
planning direction of the national heavy laboratory多模态协同认知
Paper associated data
Document Type会议论文
Affiliation1.Center for Research on Intelligent Perception and Computing, Institution of Automation, Chinese Academy of Sciences
2.School of Future Technology, University of Chinese Academy of Sciences
3.University of Adelaide
4.Center for Excellence in Brain Science and Intelligence Technology (CEBSIT)
5.Chinese Academy of Sciences, Artificial Intelligence Research (CAS-AIR)
Recommended Citation
GB/T 7714
Dong An,Yuankai Qi,Yan Huang,et al. Neighbor-view Enhanced Model for Vision and Language Navigation[C],2021.
Files in This Item: Download All
File Name/Size DocType Version Access License
nvem.pdf(2412KB)会议论文 开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Dong An]'s Articles
[Yuankai Qi]'s Articles
[Yan Huang]'s Articles
Baidu academic
Similar articles in Baidu academic
[Dong An]'s Articles
[Yuankai Qi]'s Articles
[Yan Huang]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Dong An]'s Articles
[Yuankai Qi]'s Articles
[Yan Huang]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: nvem.pdf
Format: Adobe PDF
All comments (0)
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.