Hierarchical Attention Networks for Fact-based Visual Question Answering

doi:10.1007/s11042-023-16151-w

	Hierarchical Attention Networks for Fact-based Visual Question Answering
	Yao, Haibo 1; Luo, Yongkang2 ; Zhang, Zhi 1; Yang, Jianhang 1; Cai, Chengtao 1
发表期刊	MULTIMEDIA TOOLS AND APPLICATIONS
ISSN	1380-7501
	2023-07-22
页码	18
通讯作者	Zhang, Zhi(zhangzhi1981@hrbeu.edu.cn)
摘要	Fact-based Visual Question Answering (FVQA) aims to answer questions with images and facts. It requires a fine-grained and simultaneous understanding visual content, textual questions, and factual knowledge. We propose a novel Hierarchical Attention Network (HANet) for FVQA to address the limitations of existing methods. Most existing FVQA methods only consider external facts as a library of answers, which weakens the role of the external facts, and ignore information from images, questions, and external knowledge. Additionally, they only utilize appearance features of images and disregard position information, which results in a model failing to answer many complex questions, due to the absence of important information in images. Our proposed model considers FVQA as a triple modal interaction task and exploits self-attention and multiple attention interaction to make full use of information from all three modalities. In specific, we introduce three attention modules: Self-Attention Layer, Triple-modal Attention Layer, and Bi-Attention Layer to sufficiently extract useful information from images, questions, facts. Furthermore, we also introduce positional encoding into image embedding acquisition to further improve performance of the model. Our proposed method achieves state-of-the-art performance on the FVQA dataset, with top-3 accuracy of 85.98% and top-1 accuracy of 71.68%.
关键词	Fact-based Visual Question Answering Hierarchical attention networks Self-attention Multiple attention interaction Positional encoding
DOI	10.1007/s11042-023-16151-w
关键词[WOS]	RECOMMENDATION SYSTEM ; GRAPH
收录类别	SCI
语种	英语
资助项目	National Key R amp;D Program of China[2019YFE0105400]
项目资助者	National Key R amp;D Program of China
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Computer Science, Theory & Methods ; Engineering, Electrical & Electronic
WOS记录号	WOS:001034665300003
出版者	SPRINGER
引用统计
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/53898
专题	多模态人工智能系统全国重点实验室
通讯作者	Zhang, Zhi
作者单位	1.Harbin Engn Univ, Coll Intelligent Syst Sci & Engn, Harbin 150001, Peoples R China 2.Chinese Acad Sci, Inst Automat, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Yao, Haibo,Luo, Yongkang,Zhang, Zhi,et al. Hierarchical Attention Networks for Fact-based Visual Question Answering[J]. MULTIMEDIA TOOLS AND APPLICATIONS,2023:18.
APA	Yao, Haibo,Luo, Yongkang,Zhang, Zhi,Yang, Jianhang,&Cai, Chengtao.(2023).Hierarchical Attention Networks for Fact-based Visual Question Answering.MULTIMEDIA TOOLS AND APPLICATIONS,18.
MLA	Yao, Haibo,et al."Hierarchical Attention Networks for Fact-based Visual Question Answering".MULTIMEDIA TOOLS AND APPLICATIONS (2023):18.