中国科学院自动化研究所机构知识库

In recent years, Transformer-based neuron networks have achieved great success in miscellaneous tasks, such as computer vision, natural language processing, speech recognition and multi-modal machine learning, etc., and even far outperformed traditional neural networks in some tasks. performance. In computer vision, Vision Transformer (ViT) is replacing convolutional neural networks as the main research focus in academia. However, compared with the traditional convolutional neural network, ViT is often unable to be practically used in some high-resolution or dense prediction vision tasks due to its quadratic increase in the amount of computation with the input scale. Designing computationally efficient ViT has become an important research topic. This paper explores acceleration for ViT via fixed-point quantization, including post-quantization, quantization training, and mixed-precision quantization. The main contributions of this paper are as follows:

• Aiming at the requirement of fast quantization deployment of ViT, we propose a  post-quantization approach based on minimizing the quantization error. The mean square error is used to estimate the quantization error, and an iterative method is used to solve the optimization problem. Through this quantization method, the optimal quantization step size can be quickly and accurately obtained. The performance of the 8-bit quantized model surpasses the previous work of ViT post-quantization on ImageNet.
• Aiming at the problem of excessive accuracy loss in low-bit quantization of ViT using post-quantization, we propose a differentiable quantization-aware training approach, which uses learnable quantization scales to improve ViT's performance in low-bit quantization. Moreover, we introduce a learnable offset to reduce the quantization error of the GELU activation layer. Last but not the least, a MSE-based algorithm is proposed to initialize the quantization scales and offsets. This method achieves a 4-bit quantization for ViT with accuracy drop less than 0.5\% on the ImageNet dataset.
• Aiming at the problem that quantization-aware training with uniform bit-widths still suffers too much performance loss at extreme low bits, we propose a mixed-precision quantization approach based on differentiable bit-widths, named Q-ViT. First, we utilize STE to solve the gradient truncation problem of bit-widths, enabling the joint training of bit-widths and quantization scales. Next, we propose a novel technique named switchable scales to solve the instability problem in the joint training process. Finally, we leverage head-wise bit-width for the multi-head mechanism in the self-attention process to further squeeze the size of Q-ViT. Q-ViT achieves 3-bit quantization with a mild accuracy drop on ImageNet, surpassing the previous state-of-the-art uniform quantization work.