Knowledge Commons of Institute of Automation,CAS
Hierarchical Attention Networks for Fact-based Visual Question Answering | |
Yao, Haibo1; Luo, Yongkang2![]() | |
发表期刊 | MULTIMEDIA TOOLS AND APPLICATIONS
![]() |
ISSN | 1380-7501 |
2023-07-22 | |
页码 | 18 |
通讯作者 | Zhang, Zhi(zhangzhi1981@hrbeu.edu.cn) |
摘要 | Fact-based Visual Question Answering (FVQA) aims to answer questions with images and facts. It requires a fine-grained and simultaneous understanding visual content, textual questions, and factual knowledge. We propose a novel Hierarchical Attention Network (HANet) for FVQA to address the limitations of existing methods. Most existing FVQA methods only consider external facts as a library of answers, which weakens the role of the external facts, and ignore information from images, questions, and external knowledge. Additionally, they only utilize appearance features of images and disregard position information, which results in a model failing to answer many complex questions, due to the absence of important information in images. Our proposed model considers FVQA as a triple modal interaction task and exploits self-attention and multiple attention interaction to make full use of information from all three modalities. In specific, we introduce three attention modules: Self-Attention Layer, Triple-modal Attention Layer, and Bi-Attention Layer to sufficiently extract useful information from images, questions, facts. Furthermore, we also introduce positional encoding into image embedding acquisition to further improve performance of the model. Our proposed method achieves state-of-the-art performance on the FVQA dataset, with top-3 accuracy of 85.98% and top-1 accuracy of 71.68%. |
关键词 | Fact-based Visual Question Answering Hierarchical attention networks Self-attention Multiple attention interaction Positional encoding |
DOI | 10.1007/s11042-023-16151-w |
关键词[WOS] | RECOMMENDATION SYSTEM ; GRAPH |
收录类别 | SCI |
语种 | 英语 |
资助项目 | National Key R amp;D Program of China[2019YFE0105400] |
项目资助者 | National Key R amp;D Program of China |
WOS研究方向 | Computer Science ; Engineering |
WOS类目 | Computer Science, Information Systems ; Computer Science, Software Engineering ; Computer Science, Theory & Methods ; Engineering, Electrical & Electronic |
WOS记录号 | WOS:001034665300003 |
出版者 | SPRINGER |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/53898 |
专题 | 多模态人工智能系统全国重点实验室 |
通讯作者 | Zhang, Zhi |
作者单位 | 1.Harbin Engn Univ, Coll Intelligent Syst Sci & Engn, Harbin 150001, Peoples R China 2.Chinese Acad Sci, Inst Automat, Beijing 100190, Peoples R China |
推荐引用方式 GB/T 7714 | Yao, Haibo,Luo, Yongkang,Zhang, Zhi,et al. Hierarchical Attention Networks for Fact-based Visual Question Answering[J]. MULTIMEDIA TOOLS AND APPLICATIONS,2023:18. |
APA | Yao, Haibo,Luo, Yongkang,Zhang, Zhi,Yang, Jianhang,&Cai, Chengtao.(2023).Hierarchical Attention Networks for Fact-based Visual Question Answering.MULTIMEDIA TOOLS AND APPLICATIONS,18. |
MLA | Yao, Haibo,et al."Hierarchical Attention Networks for Fact-based Visual Question Answering".MULTIMEDIA TOOLS AND APPLICATIONS (2023):18. |
条目包含的文件 | 条目无相关文件。 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论