KM4: Visual reasoning via Knowledge EmbeddingMemoryModel with MutualModulation

doi:10.1016/j.inffus.2020.10.007

CASIA OpenIR > 多模态人工智能系统全国重点实验室 > 平行智能技术与系统团队

	KM4: Visual reasoning via Knowledge EmbeddingMemoryModel with MutualModulation
	Zheng, Wenbo 1,2; Yan, Lan2,3 ; Gou, Chao4 ; Wang, Fei-Yue2
发表期刊	INFORMATION FUSION
ISSN	1566-2535
	2021-03-01
卷号	67 页码:14-28
通讯作者	Wang, Fei-Yue(feiyue.wang@ia.ac.cn)
摘要	Visual reasoning is a special kind of visual question answering, which is essentially multi-step and compositional, and also requires intensive text-visual interaction. The most important and challenging problem of visual reasoning is to design an effective and robust visual reasoning model. To this end, there are two challenges to overcome. The first is that textual and visual information must be jointly considered to make accurate inferences about reasoning. The second is that existing deep learning-based works are often too specific to a particular task. To address these issues, we propose a knowledge memory embedding model with mutual modulation for visual reasoning. This approach learns not only knowledge-based embeddings derived from key-value memory network to make the full and joint of textual and visual information, but also exploits the prior knowledge to improve the performance with knowledge-based representation learning for applying other general reasoning tasks. Experimental results on four benchmarks show that the proposed approach significantly improves performance compared with other state-of-the-art methods, guarantees the robustness with our model. Most importantly, we apply our model to four reasoning tasks, and experimentally show that our model effectively supports relational reasoning and improves performance in several tasks and datasets.
关键词	Visual reasoning Knowledge-based representation learning Memory network Knowledge embedding
DOI	10.1016/j.inffus.2020.10.007
关键词[WOS]	FUSION ; MEMORY
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China[61806198] ; National Natural Science Foundation of China[61533019] ; National Natural Science Foundation of China[U1811463] ; Key Research and Development Program of Guangzhou[202007050002] ; National Key Research and Development Program of China[2018 AAA0101502]
项目资助者	National Natural Science Foundation of China ; Key Research and Development Program of Guangzhou ; National Key Research and Development Program of China
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods
WOS记录号	WOS:000598348400003
出版者	ELSEVIER
七大方向——子方向分类	多模态智能
引用统计	被引频次：14[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/42539
专题	多模态人工智能系统全国重点实验室_平行智能技术与系统团队
通讯作者	Wang, Fei-Yue
作者单位	1.Xi An Jiao Tong Univ, Sch Software Engn, Xian 710049, Peoples R China 2.Chinese Acad Sci, Inst Automat, State Key Lab Management & Control Complex Syst, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100190, Peoples R China 4.Sun Yat Sen Univ, Sch Intelligent Syst Engn, Guangzhou 510275, Peoples R China
第一作者单位	中国科学院自动化研究所
通讯作者单位	中国科学院自动化研究所
推荐引用方式 GB/T 7714	Zheng, Wenbo,Yan, Lan,Gou, Chao,et al. KM4: Visual reasoning via Knowledge EmbeddingMemoryModel with MutualModulation[J]. INFORMATION FUSION,2021,67:14-28.
APA	Zheng, Wenbo,Yan, Lan,Gou, Chao,&Wang, Fei-Yue.(2021).KM4: Visual reasoning via Knowledge EmbeddingMemoryModel with MutualModulation.INFORMATION FUSION,67,14-28.
MLA	Zheng, Wenbo,et al."KM4: Visual reasoning via Knowledge EmbeddingMemoryModel with MutualModulation".INFORMATION FUSION 67(2021):14-28.