Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain

doi:10.1016/j.patcog.2022.108837

CASIA OpenIR > 模式识别实验室

	Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain
	Shi, Qianqian 1,3; Fan, Junsong2 ; Wang, Zuoren 1,3; Zhang, Zhaoxiang2,3
发表期刊	PATTERN RECOGNITION
ISSN	0031-3203
	2022-10-01
卷号	130 页码:14
通讯作者	Wang, Zuoren(zuorenwang@ion.ac.cn) ; Zhang, Zhaoxiang(zhaoxiang.zhang@ia.ac.cn)
摘要	Multisensory integration has attracted intense studies for decades. How to combine visual and auditory information to optimize perception and decision-making is a key question in neuroscience as well as machine learning. Inspired by the mechanisms of multisensory integration in the brain, we propose a multimodal channel-wise attention transformer (MCAT) that performs reliability-weighted integration and revises the weights allocation according to a top-down attention-like mechanism. We apply MCAT on EFLSTM neural networks for a fine-grained video bird recognition task, and on MulT neural networks for an emotion recognition task. The performance of both models is improved remarkably. Ablation study shows that the attention mechanism is indispensable for effective multisensory integration. Moreover, we found that cross-modal integration models are in accordance with the law of inverse effectiveness of multisensory integration in the brain, which reveals that our model may have mechanisms similar to those in the brain. Taken together, the results demonstrate that the brain-inspired MCAT block is effective for improving multisensory integration, providing useful clues for designing new algorithms and understanding multisensory integration in the brain. (C) 2022 Elsevier Ltd. All rights reserved.
关键词	Multisensory integration Top-down attention Multimodal transformer Fine-grained bird recognition Emotion recognition
DOI	10.1016/j.patcog.2022.108837
关键词[WOS]	NEURONAL OSCILLATIONS ; MODEL
收录类别	SCI
语种	英语
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000833526200004
出版者	ELSEVIER SCI LTD
引用统计	被引频次：12[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/49825
专题	模式识别实验室
通讯作者	Wang, Zuoren; Zhang, Zhaoxiang
作者单位	1.Chinese Acad Sci, Ctr Excellence Brain Sci & Intelligence Technol, Inst Neurosci, State Key Lab Neurosci, Shanghai 200031, Peoples R China 2.Chinese Acad Sci, Ctr Res Intelligent Percept & Comp CRIPAC, Inst Automat, Natl Lab Pattern Recognit NLPR, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Sch Future Technol, Beijing 100049, Peoples R China
通讯作者单位	模式识别国家重点实验室
推荐引用方式 GB/T 7714	Shi, Qianqian,Fan, Junsong,Wang, Zuoren,et al. Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain[J]. PATTERN RECOGNITION,2022,130:14.
APA	Shi, Qianqian,Fan, Junsong,Wang, Zuoren,&Zhang, Zhaoxiang.(2022).Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain.PATTERN RECOGNITION,130,14.
MLA	Shi, Qianqian,et al."Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain".PATTERN RECOGNITION 130(2022):14.