CASIA OpenIR  > 毕业生  > 博士学位论文
类脑脉冲神经网络模型优化方法研究
李杨
2024-05-17
Pages132
Subtype博士
Abstract

随着深度学习与高性能计算等技术的飞速发展,人工智能研究在图像识别、自然语言处理等多个领域取得了巨大突破。然而,它在处理效率、能耗控制以及模型泛化等方面的局限性日益凸显。借鉴和模仿人类大脑的结构和工作原理,以实现低能耗、高效率的智能处理,是解决上述问题的一种有效途径。脉冲神经网络对生物神经元生物物理动态过程和脉冲式信息编码与传递方式进行建模,具有生物合理性高、低能耗的优点。因此开展脉冲神经网络模型优化的研究,不仅能够促进人工智能领域的理论和技术革新,对揭示生物神经信息处理机制同样具有深远的意义。

脉冲神经网络通过使用生物神经系统的脉冲机制,能够在极低的功耗下完成复杂的信息处理任务,这一特性在追求高效计算的人工智能领域中尤为关键。然而这些优势是有代价的,特别是其固有的时间依赖性和脉冲的离散特性给优化过程带来了严峻的挑战。这使得标准的反向传播算法难以适用,增加了神经网络模型优化的复杂度与难度。本文聚焦高性能、低时延和高生物可解释性的脉冲神经网络优化方法,旨在最大化发挥脉冲网络在高效推理方面的潜力:首先,本文针对基于转换的间接优化方法,深入分析了脉冲神经网络产生转换误差的原因并实现了高效的改进。随后,为了进一步减小时间延迟,考虑生物脉冲神经元的计算特点,本文提出了高效的脉冲神经网络并行计算方法。最后,本文整合间接优化方法与直接优化方法的优势,采用基于时序高效蒸馏的混合优化方法有机融合了脉冲神经网络低延迟、低能耗和高效推理等优势。本文的主要工作和创新点归纳如下:

一、基于双稳态机制的类脑脉冲神经网络优化方法。基于转换的脉冲神经网络优化方法有效地结合了成熟的反向传播算法和脉冲神经网络的高效推理特性的优势。传统人工神经网络仅使用频率对信息进行编码,而生物神经元能够通过相位编码将时序信息编码到脉冲序列中。然而,当前大多数基于相位编码的方法容易在信息传递过程中出现相位超前和相位滞后的问题,进而导致转换后的脉冲神经网络性能损失大且时间延迟高。本文通过将相位编码与双稳态机制相结合,提出了一种基于双稳态神经元的脉冲神经网络转换方法。双稳态脉冲神经元能够在相位周期内记忆以前时间步的脉冲刺激,并在下一个相位周期内产生脉冲响应,提升了脉冲神经网络向深层传递信息的准确性,有效地解决了相位超前和相位滞后的问题。此外,通过引入突触延迟机制,本文实现了信息在脉冲残差神经网络中的同步传递。实验结果表明,本文提出的基于双稳态机制的脉冲神经网络优化方法能够实现更低的时间延迟和更高的性能。

二、基于迸发脉冲机制的类脑脉冲神经网络优化方法。利用神经元脉冲发放率实现人工神经网络中激活值的近似表达,是获取高性能脉冲神经网络的有效途径。然而,大部分方法仅对模型权重进行限制或微调,忽略了生物神经元丰富的动力学特性。为了进一步降低高性能脉冲神经网络的时间延迟,首先,本文从人工神经网络与脉冲神经网络信息传播机制的差异出发,分析了当前转换方法误差产生的根本原因。然后,受生物神经元迸发脉冲机制、侧抑制机制和脉冲间隔特性启发,本文提出了迸发脉冲转换框架、侧抑制池化以及脉冲校正方法。所提方法有效地将原本残留在神经元膜电位中的信息以迸发脉冲的形式传递出去,解决了脉冲神经网络中最大池化层经常过度放电的难点,缓解了因为失活神经元脉冲造成的输出信息逼近不准确的问题。该方法在图像分类和目标检测任务中进行的实验结果表明了本文所提方法对精确近似人工神经网络激活值以及实现高性能低延迟脉冲神经网络的有效性。

三、基于并行脉冲单元的类脑脉冲神经网络优化方法。并行计算是推动当前人工智能飞速发展的重要技术之一。脉冲神经元依靠多个时间步的迭代计算整合输入信息并进行脉冲输出,这大大限制了脉冲神经网络的训练、推理以及在硬件上的应用。现有的并行方法仅对时序输入信息进行简单的加权整合,未能充分考虑脉冲神经元的计算特性。为了进一步发挥基于代理梯度的直接优化方法在低延迟方面的优势,本文提出了并行脉冲单元。通过将脉冲神经元的漏电-积分过程和放电过程分离,并利用脉冲估计的方式保留脉冲神经元的重置过程,大幅降低神经元的脉冲活动,提高了网络的稀疏性。进一步地,为了提高脉冲网络的时序信息整合能力,本文提出了输入感知和重置感知并行脉冲单元。在静态图像、序列图像、语音识别和神经形态数据集中的实验结果表明,提出的并行脉冲单元及其变体能够有效地减少仿真时间并大幅降低神经元的脉冲活动,同时提升了在多个任务上的性能。

四、基于时序高效蒸馏的类脑脉冲神经网络优化方法。当前基于转换的间接优化方法和基于代理梯度的直接优化方法各有独特的优势。然而,大部分现有混合优化方法仅将两种优化方法简单地合并在一起,没有充分考虑人工神经网络与脉冲神经网络信息传递方式的不同。此外,代理梯度优化方法虽然替代了原本尖锐的梯度,但也引起了梯度失配的问题。本文提出了一种时序高效蒸馏的方法,通过引入蒸馏损失,将间接优化方法和直接优化方法的优势有机融合。进一步地,本文将脉冲神经网络每个时间步的输出与人工神经网络教师模型对齐,实现了模型的快速收敛和性能提升。同时,本文还提出了掩码代理梯度策略,将计算得到的代理梯度进行随机掩码,在不影响模型收敛速度的同时保证了梯度的稀疏性。实验结果表明,所提方法能够有效整合两种优化方法的优势,提高了脉冲神经网络在小时间步下的性能。

Other Abstract

With the rapid development of technologies such as deep learning and high performance computing, artificial intelligence research has made significant breakthroughs in various fields, including image recognition and natural language processing. However, its limitations in processing efficiency, energy consumption control, and model generalization are increasingly apparent. Drawing inspiration from and mimicking the structure and working principles of the human brain to achieve low-energy, efficient, intelligent processing is an effective way to address these issues. Spiking neural networks model the biophysical dynamics of biological neurons and the mode of spike-based information encoding and transmission, boasting high biological plausibility and low energy consumption. Therefore, research on optimizing spiking neural network models can promote theoretical and technological innovations in artificial intelligence and have profound significance in revealing the mechanisms of biological neural information processing.

By utilizing the spiking mechanism of biological neural systems, spiking neural networks can complete complex information processing tasks with deficient power consumption, a feature particularly critical in artificial intelligence that seeks efficient computation. However, these advantages come at a cost, especially since their inherent time dependence and the discrete nature of spikes pose severe challenges to the optimization process. This makes it difficult to apply standard backpropagation algorithms, increasing the complexity and difficulty of neural network model optimization. This paper focuses on optimization methods for spiking neural networks that are high-performance, low-latency, and highly biologically interpretable, aiming to maximize the potential of efficient inference in spiking networks. First, this paper analyzes the reasons for conversion errors in spiking neural networks targeted by conversion-based indirect optimization methods and implements efficient improvements. Subsequently, this paper proposes efficient parallel computing methods for spiking neural networks to reduce time delay further, considering the computational characteristics of biological spiking neurons. Finally, this paper integrates the advantages of indirect and direct optimization methods. It adopts a hybrid optimization approach based on temporally efficient distillation, organically combining the advantages of low latency, low energy consumption, and efficient inference in spiking neural networks. The main work and innovations of this paper are summarized as follows:

1. Brain-inspired spiking neural network optimization method based on the bistable mechanism. The conversion-based optimization method for spiking neural networks effectively combines the advantages of the mature backpropagation algorithm and the efficient inference characteristics of spiking neural networks. Traditional artificial neural networks use frequency to encode information, while biological neurons can encode temporal information into spike sequences through phase coding. However, most phase coding methods are prone to phase lead and lag issues during information transmission, leading to significant performance loss and high time delay in converted spiking neural networks. This paper proposes a conversion method for spiking neural networks based on bistable neurons by combining phase coding with the bistable mechanism. Bistable spiking neurons can remember spike stimuli from previous time steps within a phase cycle and generate spike responses in the next phase cycle, improving the accuracy of information transmission to deeper layers in spiking neural networks and effectively solving the problems of phase lead and lag. Additionally, this paper achieves synchronous information transmission in residual spiking neural networks by introducing a synaptic delay mechanism. Experimental results show that the spiking neural network optimization method based on the bistable mechanism can achieve lower time delay and higher performance.

2. Brain-inspired spiking neural network optimization method based on the burst spike mechanism. Utilizing the firing rate of neurons to approximate the activation values in artificial neural networks is an effective way to achieve high-performance spiking neural networks. However, most methods only restrict or fine-tune weights, ignoring the rich dynamics of biological neurons. To further reduce the time delay in high-performance spiking neural networks, this paper first analyzes the fundamental reasons for conversion error based on the difference in information propagation mechanisms between artificial neural networks and spiking neural networks. Inspired by the biological neuron burst spike mechanism, lateral inhibition mechanism, and inter-spike interval characteristics, this paper proposes a burst spike conversion framework, lateral inhibition pooling, and spike correction method. The proposed methods effectively transmit information originally residual in the neuron membrane potential in bursting spikes, solving the difficulty of excessive discharge in the max pooling layer of spiking neural networks and alleviating the problem of inaccurate output information approximation caused by inactive neuron spikes. Experimental results in image classification and object detection tasks demonstrate the effectiveness of the proposed methods in accurately approximating artificial neural network activation values and achieving high-performance, low-latency spiking neural networks.

3. Brain-inspired spiking neural network optimization method based on parallel spike units. Parallel computing is one of the key technologies driving the rapid development of current artificial intelligence. Spiking neurons rely on iterative calculations over multiple time steps to integrate input information and produce spikes, significantly limiting the training, inference, and hardware application of spiking neural networks. Existing parallel methods only integrate temporal input information without fully considering the computational characteristics of spiking neurons. This paper proposes parallel spike units to further leverage the advantages of direct optimization methods based on surrogate gradients in low latency. Separating the leak-integration and discharge processes of spiking neurons and using spike estimation to preserve the reset process of spiking neurons significantly reduces neuronal spike activity and improves network sparsity. Furthermore, this paper proposes input-aware and reset-aware parallel spike units to enhance the temporal information integration capability of spiking networks. Experimental results in static and sequential image datasets, speech recognition, and neuromorphic datasets show that the proposed parallel spike units and their variants can effectively reduce simulation time and significantly decrease neuronal spike activity while improving performance on multiple tasks.

4. Brain-inspired spiking neural network optimization method based on temporal efficient distillation. Current conversion-based indirect and direct optimization methods based on surrogate gradients have unique advantages. However, most existing hybrid optimization methods merge the two optimization approaches without fully considering the differences in information transmission methods between artificial neural networks and spiking neural networks. Although surrogate gradient optimization methods replace the original sharp gradients, they also cause gradient mismatch issues. This paper proposes a temporal efficient distillation method that organically combines the advantages of indirect and direct optimization methods by introducing distillation loss. Further, this paper aligns the output of spiking neural networks at each time step with the teacher artificial neural network model, achieving rapid convergence and performance improvement of the model. Moreover, this paper introduces a masked surrogate gradient strategy that randomly masks the computed surrogate gradients, ensuring gradient sparsity while not affecting the model's convergence speed. Experimental results demonstrate that the proposed method can effectively integrate the advantages of both optimization approaches, improving the performance of spiking neural networks at small time steps.

Keyword脉冲神经网络优化 人工神经网络-脉冲神经网络转换 时空反向传播 时序高效知识蒸馏 高性能低延迟
Language中文
Sub direction classification类脑模型与计算
planning direction of the national heavy laboratory认知机理与类脑学习
Document Type学位论文
Identifierhttp://ir.ia.ac.cn/handle/173211/57196
Collection毕业生_博士学位论文
Recommended Citation
GB/T 7714
李杨. 类脑脉冲神经网络模型优化方法研究[D],2024.
Files in This Item:
File Name/Size DocType Version Access License
YangLi_Thesis.pdf(7989KB)学位论文 限制开放CC BY-NC-SA
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[李杨]'s Articles
Baidu academic
Similar articles in Baidu academic
[李杨]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[李杨]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.