Pay attention to doctor & ndash;patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis
Zheng, Wenbo1,2; Yan, Lan2,3; Gou, Chao4; Zhang, Zhi-Cheng5; Zhang, Jun Jason6; Hu, Ming7; Wang, Fei-Yue2
Source PublicationINFORMATION FUSION
ISSN1566-2535
2021-11-01
Volume75Pages:168-185
Corresponding AuthorWang, Fei-Yue(feiyue.wang@ia.ac.cn)
AbstractThe sudden increase in coronavirus disease 2019 (COVID-19) cases puts high pressure on healthcare services worldwide. At this stage, fast, accurate, and early clinical assessment of the disease severity is vital. In general, there are two issues to overcome: (1) Current deep learning-based works suffer from multimodal data adequacy issues; (2) In this scenario, multimodal (e.g., text, image) information should be taken into account together to make accurate inferences. To address these challenges, we propose a multi-modal knowledge graph attention embedding for COVID-19 diagnosis. Our method not only learns the relational embedding from nodes in a constituted knowledge graph but also has access to medical knowledge, aiming at improving the performance of the classifier through the mechanism of medical knowledge attention. The experimental results show that our approach significantly improves classification performance compared to other state-of-the-art techniques and possesses robustness for each modality from multi-modal data. Moreover, we construct a new COVID-19 multi-modal dataset based on text mining, consisting of 1393 doctor-patient dialogues and their 3706 images (347 X-ray + 2598 CT + 761 ultrasound) about COVID-19 patients and 607 non-COVID-19 patient dialogues and their 10754 images (9658 X-ray + 494 CT + 761 ultrasound), and the fine-grained labels of all. We hope this work can provide insights to the researchers working in this area to shift the attention from only medical images to the doctor-patient dialogue and its corresponding medical images.
KeywordCOVID-19 diagnose Knowledge attention mechanism Knowledge-based representation learning Knowledge embedding
DOI10.1016/j.inffus.2021.05.015
WOS KeywordPREDICTING COVID-19 ; FUSION ; ACCURATE ; NETWORK ; CANCER ; SYSTEM
Indexed BySCI
Language英语
Funding ProjectNational Key R&D Program of China[2020YFB1600400] ; National Natural Science Foundation of China[61806198] ; National Natural Science Foundation of China[61533019] ; National Natural Science Foundation of China[U1811463] ; Key Technologies Research and Development Program of Guangzhou, China[202007050002] ; National Key Research and Development Program of China[2018AAA0101502]
Funding OrganizationNational Key R&D Program of China ; National Natural Science Foundation of China ; Key Technologies Research and Development Program of Guangzhou, China ; National Key Research and Development Program of China
WOS Research AreaComputer Science
WOS SubjectComputer Science, Artificial Intelligence ; Computer Science, Theory & Methods
WOS IDWOS:000671018300014
PublisherELSEVIER
Sub direction classification多模态智能
Citation statistics
Cited Times:4[WOS]   [WOS Record]     [Related Records in WOS]
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/45644
Collection复杂系统管理与控制国家重点实验室_平行智能技术与系统团队
Corresponding AuthorWang, Fei-Yue
Affiliation1.Xi An Jiao Tong Univ, Sch Software Engn, Xian 710049, Peoples R China
2.Chinese Acad Sci, Inst Automat, State Key Lab Management & Control Complex Syst, Beijing 100190, Peoples R China
3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100190, Peoples R China
4.Sun Yat Sen Univ, Sch Intelligent Syst Engn, Guangzhou 510275, Peoples R China
5.Gen Hosp Peoples Liberat Army, Med Ctr 7, Beijing 100700, Peoples R China
6.Wuhan Univ, Sch Elect Engn & Automat, Wuhan 430072, Peoples R China
7.Wuhan Pulm Hosp, Intens Care Unit, Wuhan 430030, Peoples R China
First Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Corresponding Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
Zheng, Wenbo,Yan, Lan,Gou, Chao,et al. Pay attention to doctor & ndash;patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis[J]. INFORMATION FUSION,2021,75:168-185.
APA Zheng, Wenbo.,Yan, Lan.,Gou, Chao.,Zhang, Zhi-Cheng.,Zhang, Jun Jason.,...&Wang, Fei-Yue.(2021).Pay attention to doctor & ndash;patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis.INFORMATION FUSION,75,168-185.
MLA Zheng, Wenbo,et al."Pay attention to doctor & ndash;patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis".INFORMATION FUSION 75(2021):168-185.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Zheng, Wenbo]'s Articles
[Yan, Lan]'s Articles
[Gou, Chao]'s Articles
Baidu academic
Similar articles in Baidu academic
[Zheng, Wenbo]'s Articles
[Yan, Lan]'s Articles
[Gou, Chao]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Zheng, Wenbo]'s Articles
[Yan, Lan]'s Articles
[Gou, Chao]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.