Object-difference drived graph convolutional networks for visual question answering | |
Zhu, Xi1,2; Mao, Zhendong3; Chen, Zhineng4; Li, Yangyang5; Wang, Zhaohui1,2; Wang, Bin6 | |
发表期刊 | MULTIMEDIA TOOLS AND APPLICATIONS |
ISSN | 1380-7501 |
2020-03-20 | |
页码 | 19 |
通讯作者 | Mao, Zhendong(maozhendong2008@gmail.com) |
摘要 | Visual Question Answering(VQA), an important task to evaluate the cross-modal understanding capability of an Artificial Intelligence model, has been a hot research topic in both computer vision and natural language processing communities. Recently, graph-based models have received growing interest in VQA, for its potential of modeling the relationships between objects as well as its formidable interpretability. Nonetheless, those solutions mainly define the similarity between objects as their semantical relationships, while largely ignoring the critical point that the difference between objects can provide more information for establishing the relationship between nodes in the graph. To achieve this, we propose an object-difference based graph learner, which learns question-adaptive semantic relations by calculating inter-object difference under the guidance of questions. With the learned relationships, the input image can be represented as an object graph encoded with structural dependencies between objects. In addition, existing graph-based models leverage the pre-extracted object boxes by the object detection model as node features for convenience, but they are suffering from the redundancy problem. To reduce the redundant objects, we introduce a soft-attention mechanism to magnify the question-related objects. Moreover, we incorporate our object-difference based graph learner into the soft-attention based Graph Convolutional Networks to capture question-specific objects and their interactions for answer prediction. Our experimental results on the VQA 2.0 dataset demonstrate that our model gives significantly better performance than baseline methods. |
关键词 | Visual question answering Graph convolutional networks Object-difference |
DOI | 10.1007/s11042-020-08790-0 |
收录类别 | SCI |
语种 | 英语 |
资助项目 | National Key Research and Development Program of China[2016QY03D0505] ; National Natural Science Foundation of China[U19A2057] |
项目资助者 | National Key Research and Development Program of China ; National Natural Science Foundation of China |
WOS研究方向 | Computer Science ; Engineering |
WOS类目 | Computer Science, Information Systems ; Computer Science, Software Engineering ; Computer Science, Theory & Methods ; Engineering, Electrical & Electronic |
WOS记录号 | WOS:000521018700001 |
出版者 | SPRINGER |
七大方向——子方向分类 | 智能交互 |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/38697 |
专题 | 数字内容技术与服务研究中心_远程智能医疗 |
通讯作者 | Mao, Zhendong |
作者单位 | 1.Chinese Acad Sci, Inst Informat Engn, Beijing, Peoples R China 2.Univ Chinese Acad Sci, Sch Cyber Secur, Beijing, Peoples R China 3.Univ Sci & Technol China, Hefei, Peoples R China 4.Chinese Acad Sci, Inst Automat, Beijing, Peoples R China 5.China Acad Elect & Informat Technol, Beijing, Peoples R China 6.Xiaomi Inc, Xiaomi AI Lab, Beijing, Peoples R China |
推荐引用方式 GB/T 7714 | Zhu, Xi,Mao, Zhendong,Chen, Zhineng,et al. Object-difference drived graph convolutional networks for visual question answering[J]. MULTIMEDIA TOOLS AND APPLICATIONS,2020:19. |
APA | Zhu, Xi,Mao, Zhendong,Chen, Zhineng,Li, Yangyang,Wang, Zhaohui,&Wang, Bin.(2020).Object-difference drived graph convolutional networks for visual question answering.MULTIMEDIA TOOLS AND APPLICATIONS,19. |
MLA | Zhu, Xi,et al."Object-difference drived graph convolutional networks for visual question answering".MULTIMEDIA TOOLS AND APPLICATIONS (2020):19. |
条目包含的文件 | 条目无相关文件。 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论