Knowledge Commons of Institute of Automation,CAS
Attention Spiking Neural Networks | |
Yao, Man1,2; Zhao, Guangshe1; Zhang, Hengyu3; Hu, Yifan4; Deng, Lei4; Tian, Yonghong2,5; Xu, Bo6![]() ![]() | |
发表期刊 | IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
![]() |
ISSN | 0162-8828 |
2023-08-01 | |
卷号 | 45期号:8页码:9393-9410 |
通讯作者 | Li, Guoqi(guoqi.li@ia.ac.cn) |
摘要 | Brain-inspired spiking neural networks (SNNs) are becoming a promising energy-efficient alternative to traditional artificial neural networks (ANNs). However, the performance gap between SNNs and ANNs has been a significant hindrance to deploying SNNs ubiquitously. To leverage the full potential of SNNs, in this paper we study the attention mechanisms, which can help human focus on important information. We present our idea of attention in SNNs with a multi-dimensional attention module, which infers attention weights along the temporal, channel, as well as spatial dimension separately or simultaneously. Based on the existing neuroscience theories, we exploit the attention weights to optimize membrane potentials, which in turn regulate the spiking response. Extensive experimental results on event-based action recognition and image classification datasets demonstrate that attention facilitates vanilla SNNs to achieve sparser spiking firing, better performance, and energy efficiency concurrently. In particular, we achieve top-1 accuracy of 75.92% and 77.08% on ImageNet-1 K with single/4-step Res-SNN-104, which are state-of-the-art results in SNNs. Comparedwith counterpart Res-ANN-104, the performance gap becomes -0.95/+0.21 percent and the energy efficiency is 31.8x/7.4x. To analyze the effectiveness of attention SNNs, we theoretically prove that the spiking degradation or the gradient vanishing, which usually holds in general SNNs, can be resolved by introducing the block dynamical isometry theory. We also analyze the efficiency of attention SNNs based on our proposed spiking response visualization method. Our work lights up SNN's potential as a general backbone to support various applications in the field of SNNresearch, with a great balance between effectiveness and energy efficiency. |
关键词 | Attention mechanism efficient neuromorphic inference neuromorphic computing spiking neural network |
DOI | 10.1109/TPAMI.2023.3241201 |
关键词[WOS] | CLASSIFICATION ; COMMUNICATION ; INTELLIGENCE ; MECHANISMS |
收录类别 | SCI |
语种 | 英语 |
资助项目 | Beijing Natural Science Foundation for Distinguished Young Scholars[JQ21015] ; National Key R&D Program of China[2018AAA0102600] ; National Natural Science Foundation of China[62236009] ; National Natural Science Foundation of China[61836004] ; National Natural Science Foundation of China[U22A20103] |
项目资助者 | Beijing Natural Science Foundation for Distinguished Young Scholars ; National Key R&D Program of China ; National Natural Science Foundation of China |
WOS研究方向 | Computer Science ; Engineering |
WOS类目 | Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic |
WOS记录号 | WOS:001022958600008 |
出版者 | IEEE COMPUTER SOC |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/53886 |
专题 | 复杂系统认知与决策实验室 脑图谱与类脑智能实验室 |
通讯作者 | Li, Guoqi |
作者单位 | 1.Xi An Jiao Tong Univ, Sch Automat Sci & Engn, Xian 710049, Shaanxi, Peoples R China 2.Peng Cheng Lab, Shenzhen 518066, Guangdong, Peoples R China 3.Tsinghua Univ, Tsinghua Shenzhen Int Grad Sch, Shenzhen 518071, Guangdong, Peoples R China 4.Tsinghua Univ, Dept Precis Instrument, Ctr Brain Inspired Comp Res, Beijing 100190, Peoples R China 5.Peking Univ, Inst Artificial Intelligence, Beijing 100871, Peoples R China 6.Chinese Acad Sci, Inst Automat, Beijing 100045, Peoples R China |
通讯作者单位 | 中国科学院自动化研究所 |
推荐引用方式 GB/T 7714 | Yao, Man,Zhao, Guangshe,Zhang, Hengyu,et al. Attention Spiking Neural Networks[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2023,45(8):9393-9410. |
APA | Yao, Man.,Zhao, Guangshe.,Zhang, Hengyu.,Hu, Yifan.,Deng, Lei.,...&Li, Guoqi.(2023).Attention Spiking Neural Networks.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,45(8),9393-9410. |
MLA | Yao, Man,et al."Attention Spiking Neural Networks".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 45.8(2023):9393-9410. |
条目包含的文件 | 条目无相关文件。 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论