毕业生知识库

脑-机接口（Brain-Computer Interface，BCI）是生命科学和信息技术交叉融合的一个新兴的学科领域，它能够提供大脑和外部环境之间的直接通路。其中，基于快速序列视觉呈现范式（Rapid Serial Visual Presentation，RSVP）的BCI系统在医疗康复、生活娱乐以及军事侦察等方面有广泛的应用。基于RSVP的脑机作业认知负荷高，在实际应用中往往需要长时间连续作业，易引起作业人员警觉度水平变化，进而影响工作绩效，降低系统稳定性。因此，实现警觉度准确检测和低警觉度有效预警对于提升RSVP-BCI系统的稳定性、可靠性具有重要意义。当前，警觉度检测算法精度有限，RSVP-BCI系统存在低警觉度快速唤醒需求。本课题面向连续长程目标检测脑机作业场景，开展了多模态警觉度检测系统研究。本文的主要研究内容和创新点如下：

(1) 为研究脑机作业中警觉度和生理指标的关系，本文设计了一项基于RSVP-BCI的长时程目标检测实验，采集并建立了面向长程目标检测中警觉度的脑电（Electroencephalogram, EEG）和眼电（Electroophthalmogram, EOG）多模态数据集。对所采数据进行了特征提取，采用眼睑闭合度（PERCLOS）作为警觉度的标注，从脑电中提取微分熵特征（Differential Entropy, DE），从眼电中提取眼球运动特征。通过特征分析，发现警觉度的下降会带来脑电中慢波成分的增加和快波成分的减少，脑电中事件相关电位（Event Related Potential, ERP）的响应减弱，而眼电特征中，扫视幅度是与警觉度最相关的特征，且眼电特征的乘法组合也具有识别警觉度的潜力。

(2) 针对现有警觉度检测算法精度有限，对各模态信息挖掘不充分的问题，本文提出了一种多模态警觉度检测算法（VigilanceNet）。该方法考虑每种模态的特点，对于脑电，从时域和频域提取特征，通过设置RSVP辅助任务，从脑电中学习与任务相关的特异性响应；对眼电模态，本文提出了一种基于外积操作的可学习模块，用于捕获眼电特征中的乘法关系；对于多模态的学习，本文建议将模态内和模态间的知识进行解耦学习。在自采数据集和模拟驾驶公开数据集上验证了方法的有效性，结果显示，模型在两个数据集上分别实现了0.083和0.061的被试内预测误差，比现有最好方法降低了4.6%和9.0%，同时实现了0.115和0.094的跨被试预测误差，比现有最好方法降低了1.7%和2.0%，证明模型实现了警觉度的高精度预测。

(3) 为解决RSVP-BCI系统中人员的低警觉度快速唤醒需求，本文优选了警觉度预警方法，并开发了警觉度在线检测与预警系统。本文选择了五种常见的预警手段，即闪光、声音、触感、皮肤电刺激，以及声光刺激的组合，通过对比不同刺激对警觉度和绩效的影响效果，选择了声音刺激作为预警方案。然后，搭建了一个面向RSVP目标检测任务的警觉度在线检测与预警系统。该系统采集脑电和眼电进行实时的警觉度检测，在警觉度水平过低时进行声音预警。对系统进行了在线测试，结果表明，系统在线警觉度预测均衡精度平均值为83.8%，相对于无反馈时，行为学绩效提升了29.2%，目标检测脑电解码的均衡精度提升了8.8%，证明了所设计的警觉度检测算法和预警机制在在线应用中的有效性，改善了用户在目标检测作业中的警觉度水平和绩效表现。

本文面向连续长程RSVP目标检测脑机作业中，警觉度下降继而引起工作绩效下降的问题，设计了长时程RSVP目标检测实验，采集和分析了脑电和眼电多模态数据，设计了多模态警觉度检测算法，并搭建了警觉度在线检测与预警系统，缓解了长程脑机作业中人员的绩效下降，为长程RSVP作业的稳定性提供了理论和技术基础。

Brain-computer interface (BCI) is an interdiscipline of life science and information technology, which can provide a direct pathway between the brain and the external environment. Among them, the BCI system based on rapid serial visual presentation (RSVP) has been widely used in medical, entertainment, and military. RSVP-BCI task has high mental load and often requires long-term operation in practice, which is easy to cause fluctuation of vigilance, thus affecting work performance and reducing system stability. Therefore, it is of great significance to realize accurate estimation and enhancement of vigilance for improving the stability and reliability of the RSVP-BCI system. At present, the accuracy of the vigilance estimation algorithm is limited, and the operators of the RSVP-BCI system demand fast wake-up from low vigilance. In this paper, a multimodal vigilance estimation system is developed for the long-time target detection BCI scenario. The main contributions of this paper are as follows:

(1) In order to study the relationship between vigilance and physiological indexes in BCI tasks, this paper designed a long-term target detection experiment based on RSVP-BCI. A multimodal dataset of electroencephalogram (EEG) and electroophthalmogram (EOG) for vigilance in long-term target detection was collected. PERCLOS was used as vigilance indexes, differential Entropy (DE) was extracted from EEG, and eye movements were extracted from EOG. Through feature analysis, it was found that the decrease in vigilance would lead to the increase of slow waves and the decrease of fast waves in EEG, and the event related potential (ERP) in EEG was abated. We also found that in EOG, the saccade amplitude was the most related feature to vigilance, and the multiplicative combination of the EOG features also had the potential to identify vigilance.

(2) In view of the limited accuracy of existing vigilance estimation algorithms and insufficient mining of each modality, a multimodal vigilance estimation algorithm (VigilanceNet) was proposed in this paper. This method considered the characteristics of each modality. It extracted features from the time and frequency domains of EEG and learned task-related information using RSVP auxiliary tasks. For EOG, a learnable module based on the outer product was proposed to capture the multiplicative relation in EOG features. For multimodal learning, this paper suggested decoupling intra- and inter-modality learning. The effectiveness of the proposed method was verified on the self-collected dataset and the public dataset of simulated driving. The results showed that on the two datasets, the model achieved 0.083 and 0.061 intra-subject prediction errors, which were 4.6% and 9.0% lower than the state-of-the-art methods, and achieved 0.115 and 0.094 cross-subject errors, which were 1.7% and 2.0% lower than the existing methods. The results indicated that the proposed method achieved high accuracy in vigilance estimation.

(3) To address the demand for fast wake-up from low vigilance in the RSVP-BCI system, this paper compared and selected a vigilance enhancement method and developed a vigilance online estimation and enhancement system. In this paper, five common enhancement methods were compared, namely, flashing light, tone, touch, electrical, and a combination of tone and light stimuli. Tone stimuli was selected by comparing the effects of different stimuli on vigilance and performance. Then, an online vigilance estimation and enhancement system for the RSVP target detection task was developed. The system collected EEG and EOG to estimate vigilance in real-time, and applied tone stimuli when low vigilance level was detected. The online test results showed that the balanced accuracy of the online vigilance estimation was 83.8%. Compared with the case without stimuli, the behavior performance was improved by 29.2%, and the balanced accuracy of the target detection EEG decoding was improved by 8.8%, which proved the effectiveness of the proposed vigilance estimation algorithm and enhancement mechanism in online applications, and showed the ability of the system to improve vigilance level and performance of operators in target detection.

Aiming at the problem of decreased vigilance in the long-term RSVP target detection leading to decreased work performance, this paper first designed a long-term RSVP target detection experiment to collect and analyze EEG and EOG signals, and designed a multimodal vigilance estimation algorithm. Then built an online vigilance estimation and enhancement system. This work alleviates the decline of performance in long-term BCI tasks and provides a theoretical and technical basis for the stability of long-term RSVP target detection.