CASIA OpenIR  > 类脑智能研究中心  > 神经计算与脑机交互
基于双模态脑信息融合的精细运动想象解码研究
吴晨瑶
2022-05
Pages74
Subtype硕士
Abstract

脑机接口(Brain-Computer Interface, BCI)通过脑信号直接对外设进行控制,不经由周围神经系统。运动想象(Motor Imagery, MI)由于其不需要依赖视听觉等外界刺激,是一种重要的脑机接口范式。脑电图(Electroencephalogram, EEG)因其操作简单、便携性高等特点,是广泛应用的一种脑机接口信号。在基于EEGMI-BCI中,大多研究聚焦于对不同肢体的运动想象进行解码,取得了一定的进展,但是这限制了对外部设备直观的控制。虽然想象同一肢体不同关节的运动可以在不造成认知失联的前提下对机械臂等外设提供精细控制,但是由于EEG空间分辨率较低等原因,目前基于EEG的此类精细MI解码性能限制了其应用。功能性近红外光谱(functional Near-Infrared Spectroscopy, fNIRS)因其具有较高的空间分辨率和高便携性已经被用于MI-BCI中。将EEGfNIRS两种信号进行结合,有望提升精细MI范式的解码性能。

本研究设计了一个精细MI范式,包括四个类别(手、腕、肩和静息状态),并采集了16名被试的EEGfNIRS双模态脑活动数据。通过时频分析,发现3个类别的EEG信号都存在明显的对侧占优的事件相关去同步化(Event-Related Desynchronization, ERD)现象,且在对侧运动区的α频段和β频段存在显著差异。4个类别的fNIRS信号激活模式在时间维度上存在显著差异,基于峰值的脑地形图,显示4个类别在空间上也存在一定区别。

本研究提出了一种双模态融合网络以解码精细MI任务。在该网络框架中,两个基于卷积神经网络(Convolutional Neural Networks, CNN)的特征提取器被用于提取每个模态的特征,并在特征层面上采用多模态传输模块(Multi-Modal Transfer Module, MMTM)对两个模态的特征进行融合。所提方法四分类准确率达到59.22%,与单模态网络的结果相比,所提的双模态方法可以显著提升分类性能,并且显著高于所有对比方法。

综上所述,本学位论文所验证了EEGfNIRS双模态数据在精细MI任务中的有效性,表明所提出的双模态融合方法可以提升精细MI任务的解码,本研究有望为基于MI任务的精细BCI控制系统提供技术支持。

Other Abstract

Brain-computer interface (BCI) controls external devices directly through brain signals, without relying on the peripheral nervous system. Motor imagery (MI) is an important BCI paradigm because it does not need to rely on external stimuli such as audio-visual perception. Electroencephalogram (EEG) is a widely-used brain-computer interface signal because of its simple operation and high portability. In EEG based MI-BCI, most studies focus on decoding the motion imagination of different limbs, and some progress has been made, but this limits the intuitive control of external devices. Although imagining the movement of different joints of the same limb can provide fine control of peripherals such as manipulator without cognitive loss, due to the low spatial resolution of EEG, the current fine MI decoding performance based on EEG limits its application. Functional near infrared spectroscopy (fNIRS) has been used in MI-BCI because of its high spatial resolution and high portability. Combining EEG and fNIRS signals is expected to improve the decoding performance of fine MI paradigm.

This study designed a fine MI paradigm, including four categories (hand, wrist, shoulder and rest), and collected the EEG and fNIRS bimodal brain activity data of 16 subjects. Through time-frequency analysis, it is found that there is an obvious contralateral dominant event-related desynchronization (ERD) phenomenon in the three categories of EEG signals, and there are significant differences in frequency bands α and β in the contralateral motion area. The fNIRS signal activation patterns of the four categories are significantly different in the time dimension. The brain topographic map based on the peak shows that the four categories are also different in space.

This study proposed a bimodal fusion network to improve the recognition performance of fine MI tasks. In this thesis, two feature extractors based on convolutional neural networks (CNN) are used to extract the features of each modality respectively, and the features of the two modalities are fused at feature level using Multi-Modal Transfer Module (MMTM). Compared with the results of single-modal network, the proposed bimodal method can significantly improve the classification performance, and is significantly higher than all comparison methods.

In summary, this thesis verifies the effectiveness of EEG and fNIRS bimodal data in fine MI tasks, and shows that the proposed bimodal fusion method can improve the decoding of fine MI tasks. This research is expected to provide technical support for the fine BCI control system based on MI tasks.

Keyword运动想象 多模态融合 脑机接口 深度学习
Language中文
Document Type学位论文
Identifierhttp://ir.ia.ac.cn/handle/173211/48822
Collection类脑智能研究中心_神经计算与脑机交互
毕业生_硕士学位论文
Recommended Citation
GB/T 7714
吴晨瑶. 基于双模态脑信息融合的精细运动想象解码研究[D]. 中国科学院自动化研究所. 中国科学院自动化研究所,2022.
Files in This Item:
File Name/Size DocType Version Access License
学位论文-吴晨瑶-2.pdf(9743KB)学位论文 开放获取CC BY-NC-SA
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[吴晨瑶]'s Articles
Baidu academic
Similar articles in Baidu academic
[吴晨瑶]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[吴晨瑶]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.