Knowledge Commons of Institute of Automation,CAS
Normalized and Geometry-Aware Self-Attention Network for Image Captioning | |
Guo LT(郭龙腾)1,2; Liu J(刘静)1; Zhu XX(朱欣鑫)1; Yao P(姚鹏)3; Lu SC(卢诗晨)4; Lu HQ(卢汉清)1 | |
2020 | |
会议名称 | CVPR |
会议日期 | 2020.06.14 |
会议地点 | 线上 |
摘要 | Self-attention (SA) network has shown profound value in image captioning. In this paper, we improve SA from two aspects to promote the performance of image captioning. First, we propose Normalized Self-Attention (NSA), a reparameterization of SA that brings the benefits of normalization inside SA. While normalization is previously only applied outside SA, we introduce a novel normalization method and demonstrate that it is both possible and beneficial to perform it on the hidden activations inside SA. Second, to compensate for the major limit of Transformer that it fails to model the geometry structure of the input objects, we propose a class of Geometry-aware Self-Attention (GSA) that extends SA to explicitly and efficiently consider the relative geometry relations between the objects in the image. To construct our image captioning model, we combine the two modules and apply it to the vanilla self-attention network. We extensively evaluate our proposals on MS-COCO image captioning dataset and superior results are achieved when comparing to state-of-the-art approaches. Further experiments on three challenging tasks, i.e. video captioning, machine translation, and visual question answering, show the generality of our methods. |
关键词 | Image captioning Self-attention |
收录类别 | SCI |
七大方向——子方向分类 | 多模态智能 |
文献类型 | 会议论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/44987 |
专题 | 紫东太初大模型研究中心_图像与视频分析 |
通讯作者 | Liu J(刘静) |
作者单位 | 1.National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences 2.School of Artificial Intelligence, University of Chinese Academy of Sciences 3.University of Science and Technology Beijing 4.Wuhan University |
第一作者单位 | 模式识别国家重点实验室 |
通讯作者单位 | 模式识别国家重点实验室 |
推荐引用方式 GB/T 7714 | Guo LT,Liu J,Zhu XX,et al. Normalized and Geometry-Aware Self-Attention Network for Image Captioning[C],2020. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
[CVPR 2020] Normaliz(574KB) | 会议论文 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论