4.提出基于 Unet-Transformer 的联合单张图像高光检测与去除方法。
With recent advances in developing a variety of digital hardware devices and computational photography techniques, the image has become one of the most important carriers of visual information. Images captured in real-world scenes usually have specular highlights, which to a certain extent weaken or even lose some intrinsic color and texture information of objects.
This will have a negative impact on subsequent image understanding and applications, such as image segmentation, object recognition, 3D reconstruction, etc. Therefore, detecting and removing specular highlight effects and restoring complete texture details have drawn a lot of attention in the fields of computer vision, image processing and multimedia. Importantly, the achievements of this study are also of great significance in practical applications.
The task of specular highlight detection and removal has been studied for decades, but existing knowledge-driven traditional methods and data-driven deep learning methods are still difficult to effectively remove the specular highlights in real-world scenes. In order to effectively remove the specular highlights, this thesis firstly constructs a real-world dataset for specular highlight removal. Based on this dataset, a basic framework utilizing a convolutional neural network is proposed to solve the problem of traditional algorithms that generate a large number of black shadows. Further, the local self-attention mechanism is used to iteratively perform specular highlight detection and removal to facilitate the resolution of the inaccurate restoration of texture details in highlight regions. At last, this study explores joint specular highlight detection and removal with the simultaneous existence of three light effects including reflection, refraction, and transmission.
The main contributions and innovations of this thesis are summarized as follows:
1. Construction of a real-world specular removal dataset.
There is no public specular highlight removal dataset of real-world for training highlight removal neural networks. Based on the optical theoretical foundations such as the Fresnel reflection model, Snell's law of refraction and Marius' law, we established a studio with controllable lightings for photography. By utilizing this studio, a high-quality Specular-Diffuse benchmark dataset is first obtained, in which each specular highlight image is paired with the ground-truth specular-free diffuse image. The large-scale dataset contains 13,380 images in 2,310 different scenes, which can be used for the training of deep neural models and for quantitative evaluation of specular highlight removal methods.
2. A convolutional neural network for single-image highlight removal.
This thesis proposes a new data-driven method to automatically remove specular highlights in single image. The features of images are extracted through the VGG-19 network, and the highlight masks and the HSV color space are introduced to reduce the chromatic aberration. Then, the generated adversarial network is used to judge the quality of the generated image. In order to improve the similarity between the generated non-specular image and the real diffuse image, the trained model utilizes a contextual loss function, an adversarial loss function, and a consistency loss function to jointly optimize the model parameters. The experimental results in real scenes show that the proposed method can eliminate black artifacts in specular highlight removal of a single image.
3. Single-image highlight removal based on a local self-attention mechanism.
We propose an end-to-end dual-branch network for highlight removal based on Generative Adversarial Network (GAN), in which one branch is used for specular highlight detection and the other branch is used for specular highlight removal. In detail, we iteratively perform specular highlight detection and removal tasks. By locating the highlight areas and introducing the attention mechanism, we directly model the mapping relationship between diffusion and specular highlight areas, making the network focused on the specular highlight areas distribution and texture details. Thus it enhances the effect of specular highlight removal, while reducing the differences between the generated images and ground-truth diffuse images. A large number of comparative experimental results show that this method effectively solves the problem of inaccurate restoration of texture details after highlight removal in a single image, and has better removal effects for areas with sparse distribution and small highlight areas. In addition, through the exhaustive experiments on real indoor and outdoor scenes, we show that the proposed network has strong generalization performance, and it also indicates that the dataset established in Work 1 is close to the actual lighting conditions of the real environment.
4. Joint specular highlight detection and removal in single images via Unet-Transformer.
In order to extend our algorithm to the scenes under more complex lighting conditions, we then propose a new deep neural model which jointly detects and removes specular highlights from a single image. We have observed that scenes with specular highlights in the real-world have two common characteristics: firstly, specular highlights are usually small-sized and sparsely distributed; secondly, both the colors of the highlights and the refracted light in the highlight areas are similar to the color of the light source. In this thesis, we utilize an encoder-decoder network to detect specular highlights and generate a highlight mask image. Then the two images are input into a Transformer network for highlight removal. The Swin transformer we applied works well to capture global features and establish relationships between continuous self-attention layers. This enables interaction and connection between windows of the previous layer, which greatly improves the expressive ability of the model. Through the experiments on a public benchmark dataset and real indoor and outdoor scenes, we show that our method can effectively detect and remove the specular highlight of objects such as metals and transparent materials under complex lighting conditions. Our approach further improves the accuracy and generalization performance of highlight removal.
|吴仲琦. 基于深度学习的真实场景单张图像高光去除研究[D]. 中国科学院自动化研究所. 中国科学院大学,2022.
|Files in This Item:
|Recommend this item
|Export to Endnote
|Similar articles in Google Scholar
|Similar articles in Baidu academic
|Similar articles in Bing Scholar
Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.