Knowledge Commons of Institute of Automation,CAS
Fine-grained neural decoding with distributed word representations | |
Wang, Shaonan1,2; Zhang, Jiajun1,2; Wang, Haiyan1,2,3; Lin, Nan4,5; Zong, Chengqing1,2,6 | |
发表期刊 | INFORMATION SCIENCES |
ISSN | 0020-0255 |
2020 | |
卷号 | 507页码:256-272 |
通讯作者 | Wang, Shaonan(shaonan.wang@nlpr.ia.ac.cn) |
摘要 | fMRI word decoding refers to decode what the human brain is thinking by interpreting functional Magnetic Resonance Imaging (fMRI) scans from people watching or listening to words, representing a sort of mind-reading technology. Existing works decoding words from imaging data have been largely limited to concrete nouns from a relatively small number of semantic categories. Moreover, such studies use different word-stimulus presentation paradigms and different computational models, lacking a comprehensive understanding of the influence of different factors on fMRI word decoding. In this paper, we present a large-scale evaluation of eight word embedding models and their combinations for decoding fine-grained fMRI data associated with three classes of words recorded from three stimulus-presentation paradigms. Specifically, we investigate the following research questions: (1) How does the brain-image decoder perform on different classes of words? (2) How does the brain-image decoder perform in different stimulus-presentation paradigms? (3) How well does each word embedding model allow us to decode neural activation patterns in the human brain? Furthermore, we analyze the most informative voxels associated with different classes of words, stimulus-presentation paradigms and word embedding models to explore their neural basis. The results have shown the following: (1) Different word classes can be decoded most effectively with different word embedding models. Concrete nouns and verbs are more easily distinguished than abstract nouns and verbs. (2) Among the three stimulus-presentation paradigms (picture, sentence and word clouds), the picture paradigm achieves the highest decoding accuracy, followed by the sentence paradigm. (3) Among the eight word embedding models, the model that encodes visual information obtains the best performance, followed by models that encode textual and contextual information. (4) Compared to concrete nouns, which activate mostly vision-related brain regions, abstract nouns activate broader brain regions such as the visual, language and default-mode networks. Moreover, both the picture paradigm and the model that encodes visual information have stronger associations with vision-related brain regions than other paradigms and word embedding models, respectively. (C) 2019 Elsevier Inc. All rights reserved. |
关键词 | Neural decoding fMRI word decoding Word class Stimuli paradigm Word embedding models Informative voxels |
DOI | 10.1016/j.ins.2019.08.043 |
关键词[WOS] | NATURAL IMAGES ; BRAIN ; NOUNS ; VERBS ; LANGUAGE ; REVEALS ; OBJECTS |
收录类别 | SCI |
语种 | 英语 |
资助项目 | Beijing Municipal Science and Technology Project[Z181100008918017] ; Beijing Municipal Science and Technology Project[Z181100008918017] |
项目资助者 | Beijing Municipal Science and Technology Project |
WOS研究方向 | Computer Science |
WOS类目 | Computer Science, Information Systems |
WOS记录号 | WOS:000489000500016 |
出版者 | ELSEVIER SCIENCE INC |
七大方向——子方向分类 | 自然语言处理 |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/26126 |
专题 | 多模态人工智能系统全国重点实验室_自然语言处理 |
通讯作者 | Wang, Shaonan |
作者单位 | 1.CASIA, Natl Lab Pattern Recognit, Beijing, Peoples R China 2.Univ Chinese Acad Sci, Beijing, Peoples R China 3.Chinese Acad Sci, Inst Automat, Brainnetome Ctr, Beijing, Peoples R China 4.Inst Psychol, CAS Key Lab Behav Sci, Beijing, Peoples R China 5.Univ Chinese Acad Sci, Dept Psychol, Beijing, Peoples R China 6.CAS Ctr Excellence Brain Sci & Intelligence Techn, Beijing, Peoples R China |
第一作者单位 | 模式识别国家重点实验室 |
通讯作者单位 | 模式识别国家重点实验室 |
推荐引用方式 GB/T 7714 | Wang, Shaonan,Zhang, Jiajun,Wang, Haiyan,et al. Fine-grained neural decoding with distributed word representations[J]. INFORMATION SCIENCES,2020,507:256-272. |
APA | Wang, Shaonan,Zhang, Jiajun,Wang, Haiyan,Lin, Nan,&Zong, Chengqing.(2020).Fine-grained neural decoding with distributed word representations.INFORMATION SCIENCES,507,256-272. |
MLA | Wang, Shaonan,et al."Fine-grained neural decoding with distributed word representations".INFORMATION SCIENCES 507(2020):256-272. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
1-s2.0-S002002551930(2867KB) | 期刊论文 | 作者接受稿 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论