中国科学院自动化研究所机构知识库

基于快速序列视觉呈现（Rapid Serial Visual Presentation，RSVP）的脑-机接口（Brain-Computer Interface，BCI）系统是通过对脑电（electroencephalogram，EEG）中事件相关电位（Event Related Potential，ERP）的特征提取和模式分类来实现对序列中目标图像的识别，其在医疗康复、生活娱乐以及军事侦察等方面具有巨大的应用潜力。但是，该系统在应用中存在着两方面的挑战：1）高精度解码和少校准数据相互制约：新用户在使用该系统之前，需获取大量校准数据以训练可靠的脑电解码模型，而获取大量校准数据过程十分耗时；2）人机适配性差：在系统应用中，由于用户状态的变化以及脑电非平稳的因素，模型无法保持高性能解码。本课题研究面向 RSVP-BCI 系统应用中以上两方面的难点问题，展开实现系统对新用户在小校准数据量、零校准数据情况下的高精度解码以及应用中自适应解码的算法研究。论文主要工作和创新点如下：

1. 针对 RSVP-BCI 系统在小校准数据量情况下解码性能低的问题，本文采集并建立了多被试多天的 RSVP-EEG 数据集，提出了多源条件对抗领域自适应（multi-source Conditional Adversarial Domain Adaptation，mCADA）框架。该框架包含了多个条件对抗领域自适应（CADA）网络，通过对抗训练促进网络提取目标被试（目标域）和已有被试（源域）脑电数据的共同表征，实现利用源域的有标签数据。其次，CADA 网络中的相关度量损失函数，提升同类别不同域样本之间相关性以及不同类别样本之间的差异性。在此基础上，提出了源选择策略和多源集成框架。实验结果显示，对约 5 分钟采集的校准数据，mCADA 的均衡精度可达 87.72%，与当前最先进的 EEG-Net 方法在 4 倍训练样本情况下的性能相当。因此，mCADA 框架能够实现小校准数据情况下的高精度目标检索。

2. 针对 RSVP-BCI 系统在零训练样本情况下解码性能低的问题，本文采集并建立了多被试多任务 RSVP-EEG 数据集，提出 ERP 原型匹配网络。该网络利用其他被试数据训练，将单试 EEG 和 ERP 模板映射到度量空间，并结合原型学习来获得不同被试 ERP 之间的共同表征：ERP 原型，网络按 EEG 与不同类 ERP 原型的距离对 EEG 分类。之后，设计了度量学习损失函数，按类别约束 EEG 和 ERP 原型之间的距离。此外，提出元训练策略，通过多阶段的匹配训练，提升模型泛化性能。实验结果显示，在零校准情况下，ERP 原型匹配网络的均衡精度平均可达 86.34%，与当前最先进的 EEG-Net 方法在采集约 15 分钟的校准数据情况下的性能相当；方法为 RSVP-BCI 系统的应用提供技术支持。 

3. 针对 RSVP-BCI 系统人机适配性差的问题，本文采集了多被试长时程连续 RSVP 任务 EEG 数据集，提出了自适应 ERP 原型匹配网络。该网络在面向零校准的 ERP 原型匹配网络基础上，不断加入被试的新数据及预测出的标签到训练集以更新模型。提出标签噪声学习方法，在训练迭代中调整置信度低的预测标签，缓解错误预测带来的模型性能下降。之后，引入注意力模块，训练网络能针对待分类 EEG 样本，为每个域分配合适的权重，以获得更具有表征能力的原型。实验结果表明，相比于 ERP 原型匹配网络，自适应 ERP 原型匹配网络能够一定程度上缓解 RSVP-BCI 系统长时程目标检索应用中性能的下降，为建立更为稳定长效的脑-机接口在线系统提供技术支持。

Rapid serial visual presentation (RSVP)-based Brain-computer interface (BCI) realizes target recognition by feature extraction and pattern classification of event-related potential (ERP) of electroencephalogram (EEG). The system has great potential in the application of medical rehabilitation, entertainment, and military reconnaissance. However, there are two challenges in the application of the system: 1) The high-accuracy decoding is constrained by less calibration: before using the system, new users need a time-consuming calibration process to obtain a large number of calibration data to train a reliable decoding model. 2) The user-system adaptation is poor: In the practical application of the system, due to the change of user status and non-stationary of EEG, the system can't keep high decoding performance. In this thesis, we study the above two difficult problems in the application of the RSVP-BCI system,  research on the high-accuracy decoding of new users in the case of small calibration data and zero calibration data, as well as the adaptive decoding algorithm in the application. The main work and innovation of the thesis are as follows:

1. To solve the problem of low decoding performance under the case of small calibration data in the RSVP-BCI system, we collect and establish RSVP-EEG data sets of multi-subject and multi-day, and proposes a multi-source conditional adversarial domain adaptation (mCADA) framework. Firstly, The framework contains multiple CADA networks, uses the adversarial training in the promoted CADA network to extract the common representation of source subject (domain) and target subject (domain). This makes the CADA network utilize the labelled data of the source domain. Secondly, the correlation metric learning loss function in the CADA network improve the similarity of the same category of different domains and difference of different category. Based on the CADA network, the source selection strategy and multi-source integration framework are proposed.  The experimental results show that the balanced-accuracy of the mCADA framework can reach 87.72% in the case of training samples collected in about 5 minutes, which is equivalent to the classification performance of the state-of-the-art EEG-Net in the case of 4 times of calibration data. Therefore, the mCADA framework can achieve high-accuracy target detection in the case of small calibration data.

2. Aiming at the problem of low decoding performance of the RSVP-BCI system under zero calibration, we collect and establish multi-subject and multi-task RSVP-EEG data set, and proposes ERP prototypical matching net (EPMN). The EPMN is trained on data of other subjects, EPMN maps the single-trial EEG and ERP templates to the metric space. Using prototype learning to obtain a common representation of ERP of different subjects: ERP prototype, the classification is conducted by calculating the distance between EEG and ERP prototypes. Then, a loss function of metric learning is designed to shorten the distance between EEG and ERP prototype of the same category and enlarge the distance between different categories. In addition,  a meta-training strategy is proposed, which adopts multi-stage training to improve the decoding accuracy and generalization of the model. The experimental results show that in the case of zero calibration, the average balanced-accuracy of EPMN can reach 86.34%, which is equivalent to the classification performance of the state-of-the-art EEG-Net in the case of calibration data collected in about 15 minutes. Our research provides technical support for the future application of the RSVP-BCI system.

3. To solve the problem of poor user-system adaptability of the RSVP-BCI system, we collect EEG data sets of multiple subjects in continuous long-term RSVP tasks, and proposes an adaptive ERP prototype matching network (A3EPMN). Based on the zero calibration EPMN, the A3EPMN constantly adds new data and predicted labels to the training set to update the model. A noisy-label learning method is proposed to modify the prediction labels with low confidence in the training iteration, so as to alleviate the degradation of model performance caused by false prediction. Then, the attention module is introduced to allocate appropriate weights for each domain to obtain a prototype with more representation ability for EEG samples to be classified. The experimental results show that compared with the EPMN, the A3EPMN can alleviate the decline of the classification performance to a certain extent in the long-term target retrieval of the RSVP-BCI system, and provide technical support for the establishment of a more stable and long-term BCI online system.