综合信息系统研究中心知识库

增强现实通过识别真实场景的空间结构信息，建立虚拟物体与真实场景的映射关系，智能实时地将虚拟物体放置于真实场景的恰当位置处，从而达到增强用户情境感知能力的目的。虚实遮挡技术能在虚拟物体和真实场景间建立正确的空间遮挡关系，是增强现实系统营造真实感的关键技术之一。传统增强现实系统仅能识别人工标识、图像等二维空间结构信息，简单的将虚拟物体叠加到真实场景图像上，以致真实场景始终被虚拟物体遮挡。近年来，以苹果 ARKit、谷歌ARCore、微软 HoloLens 等为代表的新一代增强现实系统采用 SLAM/VIO、深度学习等技术在相机位姿估计的同时建立真实场景的稀疏地图，能识别地图中的水平平面、垂直平面等粗粒度空间结构。其将虚拟物体放置在真实场景中的平面上，可实现较为简单的虚实遮挡效果。但对于存在非平面物体的场景，或存在弱纹理、低光照、相机快速运动的场景，其难以实现高精度的相机跟踪，且稀疏地图无法满足虚实遮挡的要求。这大大限制了增强现实的应用领域和范围。

针对上述问题，本文以 ORB-SLAM3 为基础构建增强现实系统，在获取系统的初始位姿和稀疏深度后，围绕位姿估计、稠密建图等技术内容展开了深入、系统的研究和实践，旨在提高增强现实系统实用性的同时最大限度保证虚实遮挡的鲁棒性和真实感。本文的主要工作和贡献体现在下面三个方面:

(1)提出一种改进 IMU 初始化的混合跟踪方法。针对于实际应用场景中常出现相机快速运动、弱纹理和低光照等情况导致位姿估计精度较低的问题，本文采用了视觉、惯性传感器，在最新的视觉惯性系统 ORB-SLAM3 的基础上加以改进。本文提出了一种新的面对缓慢移动的惯性传感器初始化方法加速初始化进程，同时提出一种初始化参数检测更新机制提高系统位姿估计精度。本文提出的方法在 EuRoC 数据集下进行了实验验证，主要选取了两个存在相机纯旋转、快速移动的大型复杂环境。实验表明本文的绝对轨迹误差相较 ORB-SLAM3 降低了约 10%，并远优于其他先进的视觉惯性 SLAM 系统，表明该方法有效提升了复杂场景下相机六自由度位姿估计的精准性和鲁棒性。

(2)提出了基于形态学的深度稠密化方法。在获取到图像的稀疏深度值后，本文提出了一种新的深度稠密化方法能够获取高质量的场景稠密深度图。本文先提出了一种基于形态学的深度边缘提取方法，提取形态学边缘并改进了 Canny 边缘检测算法。最终获取的深度边缘图有效削减了伪纹理边缘信息，同时保留了绝大多数的深度边缘。然后本文利用场景的稀疏深度值和深度边缘图，确定约束条件构造最优化问题进行稀疏深度传播，最终迭代求解获取具备较高质量的、边界清晰、区域平滑的场景稠密深度图。本文对于基于形态学的深度稠密化方法在 Facebook 提供的数据集下进行了定性评估和定量评估。实验均表明本文的深度稠密化效果不仅远高于基准算法，同样优于业内先进方法，展现了该深度稠密化方法较高的边缘精准性。

(3)设计并实现了增强现实原型系统。本文结合当前研究的两个方法，设计并实现了一个可交互的增强现实原型系统。用户点击选择位置，系统计算对应场景稠密深度图，通过虚拟物体和真实物体的深度值对比确认空间关系，进行虚实遮挡判定，将虚拟物体融入真实世界。系统虚实融合结果表明，本系统在数据集和真实场景中均拥有良好的虚实遮挡效果，具备较高的精准性、鲁棒性和真实感，能够满足用户应用需求，也证明了本文所提出的研究方法具备较好的实用价值。

Augmented reality recognizes the spatial structure of the real scene, establishes the mapping relationship between virtual objects and the real scene, and intelligently places virtual objects in an appropriate position of the real world to enhance users’ perception. The virtual-real occlusion technology establishes a correct spatial occlusion relationship between virtual objects and real scenes, which is one of the key technologies for augmented reality to create a sense of reality. Traditional augmented reality systems can only recognize two-dimensional spatial structures such as artificial signs, and simply superimpose virtual objects on images, so that the real scene is always masked by virtual objects. The new augmented reality systems represented by ARKit, ARCore, and HoloLens begin to use SLAM/VIO and deep learning technologies to estimate camera pose and establish the sparse map. They can identify coarse-grained spatial structures such as horizontal planes and vertical planes. By placing virtual objects on the plane of the real scene, the systems can achieve a simple virtual-real occlusion effect. However, for scenes with non-planar objects, or scenes with weak textures, low light, and rapid camera movement, it is difficult to achieve high-precision camera tracking. And sparse maps cannot meet the requirements of virtual-real occlusion, which greatly limits the application field and scope of augmented reality.

In response to the above problems, this thesis builds an augmented reality system based on ORB-SLAM3. After obtaining the initial pose and sparse depth of the system, this thesis focuses on the research of pose estimation and dense mapping technology. This thesis aims to improve the practicality of the augmented reality system while ensuring the robustness and realism of the virtual-real occlusion to the greatest extent. The main work and contributions of this thesis are as follows:

(1) Propose a hybrid tracking method with improved IMU initialization. Aiming at the problem of low pose estimation accuracy caused by rapid camera movement, weak texture, and low light in practical scenarios, this method improves on the basis of the ORB-SLAM3 system using vision sensors and inertial sensors. This paper proposes a new inertial initialization method for slow movement to speed up the initialization process, and also adds an inertial parameter update mechanism to improve the accuracy of pose estimation. This method is verified on the EuRoC dataset and mainly selected two complex environments with purely rotating and fast-moving cameras. Experiments show that the trajectory error of this thesis is about 10% lower than ORB-SLAM3, and is far better than other advanced systems, which proves that this method effectively improves the accuracy and robustness of the camera pose estimation in complex scenes.

(2) Propose a depth densification method based on morphology. After obtaining the sparse depth value of the image, this thesis proposes a new depth densification method to obtain high-quality dense depth maps. This paper first proposes a deep edge extraction method that extracts morphological edges and improves the Canny edge detection algorithm. The final depth edge remains most of the depth edges with the fewer texture edges. Then this thesis constructs an optimization problem for sparse depth propagation using the sparse depth value and depth edges of the scene. By iterative solution, the final dense depth map has clear boundaries and smooth regions. In this paper, a qualitative and quantitative evaluation of the depth densification method is carried out under the dataset provided by Facebook. Experiments show that this method is better than the benchmark algorithm and the advanced method, demonstrating the high edge accuracy of the depth densification method.

(3) Construct an augmented reality system. This thesis designs and implements an augmented reality system by combining the methods above. The user clicks to select a location, the system calculates the dense depth map of the corresponding scene and judges the virtual-real occlusion by comparing the depth values of the virtual object and the real object to place the virtual model in the real world. The results show that the system has a good virtual-real occlusion effect in the dataset and the real scene. It has high accuracy, robustness, and realism, which can meet the needs of users. It also proves the methods proposed in this thesis have good practical value.