Knowledge-Embedded Mutual Guidance for Visual Reasoning

doi:10.1109/TCYB.2023.3310892

	Knowledge-Embedded Mutual Guidance for Visual Reasoning
	Zheng, Wenbo 1,2; Yan, Lan3,4 ; Chen, Long5,6 ; Li, Qiang7 ; Wang, Fei-Yue7
发表期刊	IEEE TRANSACTIONS ON CYBERNETICS
ISSN	2168-2267
	2023-09-20
页码	13
通讯作者	Zheng, Wenbo(zwb2022@whut.edu.cn)
摘要	Visual reasoning between visual images and natural language is a long-standing challenge in computer vision. Most of the methods aim to look for answers to questions only on the basis of the analysis of the offered questions and images. Other approaches treat knowledge graphs as flattened tables to search for the answer. However, there are two major problems with these works: 1) the model disregards the fact that the world we surrounding us interlinks our hearing and speaking of natural language and 2) the model largely ignores the structure of the KG. To overcome these challenging deficiencies, a model should jointly consider two modalities of vision and language, as well as the rich structural and logical information embedded in knowledge graphs. To this end, we propose a general joint representation learning framework for visual reasoning, namely, knowledge-embedded mutual guidance. It realizes mutual guidance not only between visual data and natural language descriptions but also between knowledge graphs and reasoning models. In addition, it exploits the knowledge derived from the reasoning model to boost knowledge graphs when applying the visual relation detection task. The experimental results demonstrate that the proposed approach performs dramatically better than state-of-the-art methods on two benchmarks for visual reasoning.
关键词	Attention model joint learning knowledge embedding visual reasoning
DOI	10.1109/TCYB.2023.3310892
收录类别	SCI
语种	英语
资助项目	Natural Science Foundation of China[62303361] ; Natural Science Foundation of China[62302161] ; Natural Science Foundation of China[U1811463] ; Hainan Provincial Natural Science Foundation of China[623QN266] ; Fundamental Research Funds for the Central Universities[WUT: 233110002] ; National Key Research and Development Program of China[2018AAA0101502]
项目资助者	Natural Science Foundation of China ; Hainan Provincial Natural Science Foundation of China ; Fundamental Research Funds for the Central Universities ; National Key Research and Development Program of China
WOS研究方向	Automation & Control Systems ; Computer Science
WOS类目	Automation & Control Systems ; Computer Science, Artificial Intelligence ; Computer Science, Cybernetics
WOS记录号	WOS:001071913500001
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/53141
专题	多模态人工智能系统全国重点实验室
通讯作者	Zheng, Wenbo
作者单位	1.Wuhan Univ Technol, Sch Comp Sci & Artificial Intelligence, Wuhan 430070, Peoples R China 2.Wuhan Univ Technol, Sanya Sci & Educ Innovat Pk, Sanya 572000, Peoples R China 3.Hunan Univ, Coll Comp Sci & Engn, Changsha 410082, Peoples R China 4.Natl Supercomp Ctr Changsha, Changsha 410082, Hunan, Peoples R China 5.Chinese Acad Sci, Inst Automat, State Key Lab Management & Control Complex Syst, Beijing 100864, Peoples R China 6.Waytous Inc, Beijing 100864, Peoples R China 7.Chinese Acad Sci, Inst Automat, State Key Lab Management & Control Complex Syst, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Zheng, Wenbo,Yan, Lan,Chen, Long,et al. Knowledge-Embedded Mutual Guidance for Visual Reasoning[J]. IEEE TRANSACTIONS ON CYBERNETICS,2023:13.
APA	Zheng, Wenbo,Yan, Lan,Chen, Long,Li, Qiang,&Wang, Fei-Yue.(2023).Knowledge-Embedded Mutual Guidance for Visual Reasoning.IEEE TRANSACTIONS ON CYBERNETICS,13.
MLA	Zheng, Wenbo,et al."Knowledge-Embedded Mutual Guidance for Visual Reasoning".IEEE TRANSACTIONS ON CYBERNETICS (2023):13.