Knowledge Commons of Institute of Automation,CAS
Toward Accurate Binarized Neural Networks With Sparsity for Mobile Application | |
Wang, Peisong; He, Xiangyu; Cheng, Jian | |
发表期刊 | IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS |
ISSN | 2162-237X |
2022-05-27 | |
页码 | 13 |
通讯作者 | Cheng, Jian(jcheng@nlpria.ac.cn) |
摘要 | While binarized neural networks (BNNs) have attracted great interest, popular approaches proposed so far mainly exploit the symmetric sign function for feature binarization, i.e., to binarize activations into -1 and +1 with a fixed threshold of 0. However, whether this option is optimal has been largely overlooked. In this work, we propose the Sparsity-inducing BNN (Si-BNN) to quantize the activations to be either 0 or +1, which better approximates ReLU using 1-bit. We further introduce trainable thresholds into the backward function of binarization to guide the gradient propagation. Our method dramatically outperforms the current state-of-the-art, lowering the performance gap between full-precision networks and BNNs on mainstream architectures, achieving the new state-of-the-art on binarized AlexNet (Top-1 50.5%), ResNet-18 (Top-1 62.2%), and ResNet-50 (Top-1 68.3%). At inference time, Si-BNN still enjoys the high efficiency of bit-wise operations. In our implementation, the running time of binary AlexNet on the CPU can be competitive with the popular GPU-based deep learning framework. |
关键词 | Quantization (signal) Deep learning Convolution Training Biological neural networks Optimization Neurons Acceleration binarized neural networks (BNNs) compression fixed-point quantization |
DOI | 10.1109/TNNLS.2022.3173498 |
收录类别 | SCI |
语种 | 英语 |
资助项目 | National Key Research and Development Program of China[2021ZD0201504] ; National Natural Science Foundation of China[61906193] ; Strategic Priority Research Program of Chinese Academy of Sciences[XDA27040300] |
项目资助者 | National Key Research and Development Program of China ; National Natural Science Foundation of China ; Strategic Priority Research Program of Chinese Academy of Sciences |
WOS研究方向 | Computer Science ; Engineering |
WOS类目 | Computer Science, Artificial Intelligence ; Computer Science, Hardware & Architecture ; Computer Science, Theory & Methods ; Engineering, Electrical & Electronic |
WOS记录号 | WOS:000805801000001 |
出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/49510 |
专题 | 复杂系统认知与决策实验室_高效智能计算与学习 |
通讯作者 | Cheng, Jian |
作者单位 | Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China |
第一作者单位 | 模式识别国家重点实验室 |
通讯作者单位 | 模式识别国家重点实验室 |
推荐引用方式 GB/T 7714 | Wang, Peisong,He, Xiangyu,Cheng, Jian. Toward Accurate Binarized Neural Networks With Sparsity for Mobile Application[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2022:13. |
APA | Wang, Peisong,He, Xiangyu,&Cheng, Jian.(2022).Toward Accurate Binarized Neural Networks With Sparsity for Mobile Application.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,13. |
MLA | Wang, Peisong,et al."Toward Accurate Binarized Neural Networks With Sparsity for Mobile Application".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS (2022):13. |
条目包含的文件 | 条目无相关文件。 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论