Integrative interaction of emotional speech in audio-visual modality

doi:10.3389/fnins.2022.797277

	Integrative interaction of emotional speech in audio-visual modality
	Dong, Haibin 1; Li, Na1 ; Fan, Lingzhong2 ; Wei, Jianguo 1; Xu, Junhai 1
发表期刊	FRONTIERS IN NEUROSCIENCE
	2022-11-11
卷号	16 页码:13
通讯作者	Xu, Junhai(jhxu@tju.edu.cn)
摘要	Emotional clues are always expressed in many ways in our daily life, and the emotional information we receive is often represented by multiple modalities. Successful social interactions require a combination of multisensory cues to accurately determine the emotion of others. The integration mechanism of multimodal emotional information has been widely investigated. Different brain activity measurement methods were used to determine the location of brain regions involved in the audio-visual integration of emotional information, mainly in the bilateral superior temporal regions. However, the methods adopted in these studies are relatively simple, and the materials of the study rarely contain speech information. The integration mechanism of emotional speech in the human brain still needs further examinations. In this paper, a functional magnetic resonance imaging (fMRI) study was conducted using event-related design to explore the audio-visual integration mechanism of emotional speech in the human brain by using dynamic facial expressions and emotional speech to express emotions of different valences. Representational similarity analysis (RSA) based on regions of interest (ROIs), whole brain searchlight analysis, modality conjunction analysis and supra-additive analysis were used to analyze and verify the role of relevant brain regions. Meanwhile, a weighted RSA method was used to evaluate the contributions of each candidate model in the best fitted model of ROIs. The results showed that only the left insula was detected by all methods, suggesting that the left insula played an important role in the audio-visual integration of emotional speech. Whole brain searchlight analysis, modality conjunction analysis and supra-additive analysis together revealed that the bilateral middle temporal gyrus (MTG), right inferior parietal lobule and bilateral precuneus might be involved in the audio-visual integration of emotional speech from other aspects.
关键词	audio-visual integration emotional speech fMRI left insula weighted RSA
DOI	10.3389/fnins.2022.797277
关键词[WOS]	SUPERIOR TEMPORAL SULCUS ; HUMAN BRAIN ; PERCEPTION ; FACE ; INFORMATION ; EXPRESSIONS ; ACTIVATION ; PRECUNEUS ; INSULA ; VOICE
收录类别	SCI
语种	英语
资助项目	National Natural Science Foundation of China ; China Postdoctoral Science Foundation ; Project of Qinghai Science and Technology Program ; [62176181] ; [2020M680905] ; [2022-ZJ-T05]
项目资助者	National Natural Science Foundation of China ; China Postdoctoral Science Foundation ; Project of Qinghai Science and Technology Program
WOS研究方向	Neurosciences & Neurology
WOS类目	Neurosciences
WOS记录号	WOS:000890344700001
出版者	FRONTIERS MEDIA SA
引用统计	被引频次：2[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/51291
专题	脑图谱与类脑智能实验室_脑网络组研究
通讯作者	Xu, Junhai
作者单位	1.Tianjin Univ, Coll Intelligence & Comp, Tianjin Key Lab Cognit Comp & Applicat, Tianjin, Peoples R China 2.Chinese Acad Sci, Inst Automat, Brainnetome Ctr, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Dong, Haibin,Li, Na,Fan, Lingzhong,et al. Integrative interaction of emotional speech in audio-visual modality[J]. FRONTIERS IN NEUROSCIENCE,2022,16:13.
APA	Dong, Haibin,Li, Na,Fan, Lingzhong,Wei, Jianguo,&Xu, Junhai.(2022).Integrative interaction of emotional speech in audio-visual modality.FRONTIERS IN NEUROSCIENCE,16,13.
MLA	Dong, Haibin,et al."Integrative interaction of emotional speech in audio-visual modality".FRONTIERS IN NEUROSCIENCE 16(2022):13.