Knowledge Commons of Institute of Automation,CAS
BViT: Broad Attention-Based Vision Transformer | |
Nannan Li1,2![]() ![]() ![]() ![]() ![]() ![]() | |
发表期刊 | IEEE Transactions on Neural Networks and Learning Systems
![]() |
ISSN | 2162-237X |
2023-05 | |
页码 | 1 - 12 |
摘要 | Recent works have demonstrated that transformer can achieve promising performance in computer vision, by exploiting the relationship among image patches with self-attention. They only consider the attention in a single feature layer, but ignore the complementarity of attention in different layers. In this article, we propose broad attention to improve the performance by incorporating the attention relationship of different layers for vision transformer (ViT), which is called BViT. The broad attention is implemented by broad connection and parameter-free attention. Broad connection of each transformer layer promotes the transmission and integration of information for BViT. Without introducing additional trainable parameters, parameter-free attention jointly focuses on the already available attention information in different layers for extracting useful information and building their relationship. Experiments on image classification tasks demonstrate that BViT delivers superior accuracy of 75.0%/81.6% top-1 accuracy on ImageNet with 5M/22M parameters. Moreover, we transfer BViT to downstream object recognition benchmarks to achieve 98.9% and 89.9% on CIFAR10 and CIFAR100, respectively, that exceed ViT with fewer parameters. For the generalization test, the broad attention in Swin Transformer, T2T-ViT and LVT also brings an improvement of more than 1%. To sum up, broad attention is promising to promote the performance of attention-based models. Code and pretrained models are available at https://github.com/DRL/BViT. |
关键词 | Broad attention broad connection image classification parameter-free attention vision transformer |
DOI | 10.1109/TNNLS.2023.3264730 |
URL | 查看原文 |
收录类别 | SCI |
语种 | 英语 |
WOS记录号 | WOS:000982500800001 |
七大方向——子方向分类 | 机器学习 |
国重实验室规划方向分类 | 智能计算与学习 |
是否有论文关联数据集需要存交 | 否 |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/52190 |
专题 | 多模态人工智能系统全国重点实验室_深度强化学习 |
作者单位 | 1.The State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences 2.School of artificial intelligence, University of Chinese Academy of Sciences 3.The National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences |
第一作者单位 | 中国科学院自动化研究所 |
推荐引用方式 GB/T 7714 | Nannan Li,Yaran Chen,Weifan Li,et al. BViT: Broad Attention-Based Vision Transformer[J]. IEEE Transactions on Neural Networks and Learning Systems,2023:1 - 12. |
APA | Nannan Li,Yaran Chen,Weifan Li,Zixiang Ding,Dongbin Zhao,&Shuai Nie.(2023).BViT: Broad Attention-Based Vision Transformer.IEEE Transactions on Neural Networks and Learning Systems,1 - 12. |
MLA | Nannan Li,et al."BViT: Broad Attention-Based Vision Transformer".IEEE Transactions on Neural Networks and Learning Systems (2023):1 - 12. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
BViT_Broad_Attention(2171KB) | 期刊论文 | 作者接受稿 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论