Knowledge Commons of Institute of Automation,CAS
Global-Attention-Based Neural Networks for Vision Language Intelligence | |
Pei Liu; Yingjie Zhou; Dezhong Peng; Dapeng Wu | |
发表期刊 | IEEE/CAA Journal of Automatica Sinica |
ISSN | 2329-9266 |
2021 | |
卷号 | 8期号:7页码:1243-1252 |
摘要 | In this paper, we develop a novel global-attention-based neural network (GANN) for vision language intelligence, specifically, image captioning (language description of a given image). As many previous works, the encoder-decoder framework is adopted in our proposed model, in which the encoder is responsible for encoding the region proposal features and extracting global caption feature based on a specially designed module of predicting the caption objects, and the decoder generates captions by taking the obtained global caption feature along with the encoded visual features as inputs for each attention head of the decoder layer. The global caption feature is introduced for the purpose of exploring the latent contributions of region proposals for image captioning, and further helping the decoder better focus on the most relevant proposals so as to extract more accurate visual feature in each time step of caption generation. Our GANN is implemented by incorporating the global caption feature into the attention weight calculation phase in the word predication process in each head of the decoder layer. In our experiments, we qualitatively analyzed the proposed model, and quantitatively evaluated several state-of-the-art schemes with GANN on the MS-COCO dataset. Experimental results demonstrate the effectiveness of the proposed global attention mechanism for image captioning. |
关键词 | Global attention image captioning latent contribution |
DOI | 10.1109/JAS.2020.1003402 |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/44579 |
专题 | 学术期刊_IEEE/CAA Journal of Automatica Sinica |
推荐引用方式 GB/T 7714 | Pei Liu,Yingjie Zhou,Dezhong Peng,et al. Global-Attention-Based Neural Networks for Vision Language Intelligence[J]. IEEE/CAA Journal of Automatica Sinica,2021,8(7):1243-1252. |
APA | Pei Liu,Yingjie Zhou,Dezhong Peng,&Dapeng Wu.(2021).Global-Attention-Based Neural Networks for Vision Language Intelligence.IEEE/CAA Journal of Automatica Sinica,8(7),1243-1252. |
MLA | Pei Liu,et al."Global-Attention-Based Neural Networks for Vision Language Intelligence".IEEE/CAA Journal of Automatica Sinica 8.7(2021):1243-1252. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
JAS-2020-0373.pdf(24864KB) | 期刊论文 | 出版稿 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论