中国科学院自动化研究所机构知识库

Brain-machine interface (BMI), as a multidisciplinary technology of brain science, neuroscience, signal detection and processing, pattern recognition, etc., has received wide attention from researchers. BMI refers to the establishment of direct information channels between the brain and external devices, to achieve information interaction and functional integration between the nervous system and external devices. This thesis  presents a flexible electrodes implantation robotic system for invasive brain-machine interface applications by focusing on the key techniques of automatic focusing, color recovery, recognition and localization , pose measurement and alignment control during flexible electrodes implantation, and successfully implant the flexible electrodes into the rat cranial cortex along the cranial micro-holes in the experiment. The main research work and contributions of this thesis   are as follows:

(1) The hardware platform of the implantation robotic system and the visual interaction software interface are built, which include the functions of microscopic measurement information reading, implantation unit control and system status reading. The method of fast focusing of implantation needle, flexible electrodes and cranial micro-holes is proposed based on the focused planes intersection constraint of binocular microscope, which can effectively improve the operation efficiency of the implantation robot. For the serious color bias problem in coaxial light illumination of microscopic vision, a microscopic image color recovery method based on deep convolutional network is proposed, which can realize the color recovery of images under over-exposure, low light and different magnification conditions and effectively improve image quality.

(2) For microscopic vision scenes where the target is susceptible to light changes, out-of-focus, magnification changes and occlusions, this thesis  studies the combination of ResNet and U-shaped networks to extract multi-scale fusion features, and introduce the attention mechanism to increase the model's ability to focus on key regions and channels, and realize adaptive spatial sampling for shape perception through deformable convolution, which can effectively increase the model's ability to adapt to target shape and size. The proposed deep convolutional neural network model can simultaneously perform keypoint localization and angle estimation for the implantation needle and flexible electrode, and the results can be used for servo control of the implantation robot. To address the problem that the implantation needle easily moves out of the narrow field-of-view (FOV) of the microscope camera during the operation of the implantation robot, this thesis  focuses on the mapping relationship between the microscopic measurement unit and the implantation unit, and use the Levenberg-Marquard (LM) algorithm to solve the problem and establish the FOV following model between the microscopic measurement unit and the implantation unit to ensure that the implantation needle is always within the FOV of the microscopic measurement unit.

 (3) A method of normal vector measurement of cranial micro-holes is proposed based on polar lines and geometry constraints. The micro-hole normal vector is projected into two microscopic image spaces for pose alignment control of the implantation needle and the cranial micro-holes. Meanwhile, considering the global FOV of the implantation operation and the target details, the image Jacobian matrices at different magnifications are calibrated based on the lens magnification, motion driver and target feature size information for fast approach and accurate alignment of the implantation needle and the cranial micro-hole. Based on the developed prototype of the flexible electrodes implantation robotic system, the flexible electrodes can be successfully implanted into the mouse cranial brain, cranial micro-holes model and rat cranial micro-holes, which verifies the effectiveness of the proposed pose measurement and alignment control method.