Pay attention to doctor & ndash;patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis

doi:10.1016/j.inffus.2021.05.015

CASIA OpenIR > 多模态人工智能系统全国重点实验室 > 平行智能技术与系统团队

	Pay attention to doctor & ndash;patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis
	Zheng, Wenbo 1,2; Yan, Lan2,3 ; Gou, Chao4 ; Zhang, Zhi-Cheng 5; Zhang, Jun Jason 6; Hu, Ming 7; Wang, Fei-Yue2
发表期刊	INFORMATION FUSION
ISSN	1566-2535
	2021-11-01
卷号	75 页码:168-185
通讯作者	Wang, Fei-Yue(feiyue.wang@ia.ac.cn)
摘要	The sudden increase in coronavirus disease 2019 (COVID-19) cases puts high pressure on healthcare services worldwide. At this stage, fast, accurate, and early clinical assessment of the disease severity is vital. In general, there are two issues to overcome: (1) Current deep learning-based works suffer from multimodal data adequacy issues; (2) In this scenario, multimodal (e.g., text, image) information should be taken into account together to make accurate inferences. To address these challenges, we propose a multi-modal knowledge graph attention embedding for COVID-19 diagnosis. Our method not only learns the relational embedding from nodes in a constituted knowledge graph but also has access to medical knowledge, aiming at improving the performance of the classifier through the mechanism of medical knowledge attention. The experimental results show that our approach significantly improves classification performance compared to other state-of-the-art techniques and possesses robustness for each modality from multi-modal data. Moreover, we construct a new COVID-19 multi-modal dataset based on text mining, consisting of 1393 doctor-patient dialogues and their 3706 images (347 X-ray + 2598 CT + 761 ultrasound) about COVID-19 patients and 607 non-COVID-19 patient dialogues and their 10754 images (9658 X-ray + 494 CT + 761 ultrasound), and the fine-grained labels of all. We hope this work can provide insights to the researchers working in this area to shift the attention from only medical images to the doctor-patient dialogue and its corresponding medical images.
关键词	COVID-19 diagnose Knowledge attention mechanism Knowledge-based representation learning Knowledge embedding
DOI	10.1016/j.inffus.2021.05.015
关键词[WOS]	PREDICTING COVID-19 ; FUSION ; ACCURATE ; NETWORK ; CANCER ; SYSTEM
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China[2020YFB1600400] ; National Natural Science Foundation of China[61806198] ; National Natural Science Foundation of China[61533019] ; National Natural Science Foundation of China[U1811463] ; Key Technologies Research and Development Program of Guangzhou, China[202007050002] ; National Key Research and Development Program of China[2018AAA0101502]
项目资助者	National Key R&D Program of China ; National Natural Science Foundation of China ; Key Technologies Research and Development Program of Guangzhou, China ; National Key Research and Development Program of China
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods
WOS记录号	WOS:000671018300014
出版者	ELSEVIER
七大方向——子方向分类	多模态智能
引用统计	被引频次：29[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/45644
专题	多模态人工智能系统全国重点实验室_平行智能技术与系统团队
通讯作者	Wang, Fei-Yue
作者单位	1.Xi An Jiao Tong Univ, Sch Software Engn, Xian 710049, Peoples R China 2.Chinese Acad Sci, Inst Automat, State Key Lab Management & Control Complex Syst, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100190, Peoples R China 4.Sun Yat Sen Univ, Sch Intelligent Syst Engn, Guangzhou 510275, Peoples R China 5.Gen Hosp Peoples Liberat Army, Med Ctr 7, Beijing 100700, Peoples R China 6.Wuhan Univ, Sch Elect Engn & Automat, Wuhan 430072, Peoples R China 7.Wuhan Pulm Hosp, Intens Care Unit, Wuhan 430030, Peoples R China
第一作者单位	中国科学院自动化研究所
通讯作者单位	中国科学院自动化研究所
推荐引用方式 GB/T 7714	Zheng, Wenbo,Yan, Lan,Gou, Chao,et al. Pay attention to doctor & ndash;patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis[J]. INFORMATION FUSION,2021,75:168-185.
APA	Zheng, Wenbo.,Yan, Lan.,Gou, Chao.,Zhang, Zhi-Cheng.,Zhang, Jun Jason.,...&Wang, Fei-Yue.(2021).Pay attention to doctor & ndash;patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis.INFORMATION FUSION,75,168-185.
MLA	Zheng, Wenbo,et al."Pay attention to doctor & ndash;patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis".INFORMATION FUSION 75(2021):168-185.