Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain | |
Shi, Qianqian1,3; Fan, Junsong2![]() ![]() | |
发表期刊 | PATTERN RECOGNITION
![]() |
ISSN | 0031-3203 |
2022-10-01 | |
卷号 | 130页码:14 |
通讯作者 | Wang, Zuoren(zuorenwang@ion.ac.cn) ; Zhang, Zhaoxiang(zhaoxiang.zhang@ia.ac.cn) |
摘要 | Multisensory integration has attracted intense studies for decades. How to combine visual and auditory information to optimize perception and decision-making is a key question in neuroscience as well as machine learning. Inspired by the mechanisms of multisensory integration in the brain, we propose a multimodal channel-wise attention transformer (MCAT) that performs reliability-weighted integration and revises the weights allocation according to a top-down attention-like mechanism. We apply MCAT on EFLSTM neural networks for a fine-grained video bird recognition task, and on MulT neural networks for an emotion recognition task. The performance of both models is improved remarkably. Ablation study shows that the attention mechanism is indispensable for effective multisensory integration. Moreover, we found that cross-modal integration models are in accordance with the law of inverse effectiveness of multisensory integration in the brain, which reveals that our model may have mechanisms similar to those in the brain. Taken together, the results demonstrate that the brain-inspired MCAT block is effective for improving multisensory integration, providing useful clues for designing new algorithms and understanding multisensory integration in the brain. (C) 2022 Elsevier Ltd. All rights reserved. |
关键词 | Multisensory integration Top-down attention Multimodal transformer Fine-grained bird recognition Emotion recognition |
DOI | 10.1016/j.patcog.2022.108837 |
关键词[WOS] | NEURONAL OSCILLATIONS ; MODEL |
收录类别 | SCI |
语种 | 英语 |
WOS研究方向 | Computer Science ; Engineering |
WOS类目 | Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic |
WOS记录号 | WOS:000833526200004 |
出版者 | ELSEVIER SCI LTD |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/49825 |
专题 | 模式识别实验室 |
通讯作者 | Wang, Zuoren; Zhang, Zhaoxiang |
作者单位 | 1.Chinese Acad Sci, Ctr Excellence Brain Sci & Intelligence Technol, Inst Neurosci, State Key Lab Neurosci, Shanghai 200031, Peoples R China 2.Chinese Acad Sci, Ctr Res Intelligent Percept & Comp CRIPAC, Inst Automat, Natl Lab Pattern Recognit NLPR, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Sch Future Technol, Beijing 100049, Peoples R China |
通讯作者单位 | 模式识别国家重点实验室 |
推荐引用方式 GB/T 7714 | Shi, Qianqian,Fan, Junsong,Wang, Zuoren,et al. Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain[J]. PATTERN RECOGNITION,2022,130:14. |
APA | Shi, Qianqian,Fan, Junsong,Wang, Zuoren,&Zhang, Zhaoxiang.(2022).Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain.PATTERN RECOGNITION,130,14. |
MLA | Shi, Qianqian,et al."Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain".PATTERN RECOGNITION 130(2022):14. |
条目包含的文件 | 条目无相关文件。 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论