中国科学院自动化研究所机构知识库

近年来随着经济的飞速发展和生活水平的提高，人们对妆容的需求越来越大，促使美妆产业蓬勃发展。由于互联网上的美妆风格千人千面，线上线下的美妆产品种类众多，如何从妆容参考图像中快速精准地找到个性化的美妆产品逐渐成为研究热点。作为虚拟上妆系统的核心技术——妆容迁移技术，也受到越来越多的关注，该技术旨在将参考妆容迁移到素颜人脸上，并保持其上妆风格。该任务的主要难点包括：1）真实同人异妆数据的缺失；2）人与人的面部结构差异；3）迁移妆容样式单一。
为此，论文针对同人异妆生成问题，提出了一种多通路的分区域快速妆容迁移深度网络模型，结合妆容高层特征的语义信息优化网络。针对人与人之间的面部差异问题，分别建立上妆和卸妆对偶网络，并在此基础上提出一种美妆生成对抗网络模型，能够保持原有人脸的结构特征。针对妆容样式单一的问题，根据眼妆、唇妆、粉底妆容的差异，提出一种妆容区域划分方法，降低网络训练优化的时间，提高妆容特征的表达能力。以上的妆容迁移算法用于备选妆容生成，具有存储空间小，生成速度快的优点，在保证人脸结构不变的同时，使得迁移后的眼影更均衡，唇膏色彩更保真，粉底迁移更精细。在国际通用美妆数据库上进行实验研究，取得了更协调的视觉效果，更快的上妆速度，更多样的同人异妆和异人同妆的迁移风格，优于对比方法。
本文的主要研究工作和贡献如下：
1. 提出了一种多通路的分区域快速妆容迁移深度网络模型（FMaT）。首先在人脸关键点检测的基础上，完成端到端的人脸校准；再利用通路差异的损失函数，结合妆容高层特征的语义信息优化网络；最后通过泊松融合，将多通路的输出生成上妆结果。
2. 提出一种美妆生成对抗网络模型（BelleGAN）。整个模型只需要一个上妆网络和一个卸妆网络，能够完成素颜图像域和妆容图像域的相互转化，两个网络相互博弈地训练。并利用对抗损失、循环一致损失和妆容损失优化网络。
3. 提出一种妆容区域划分方法，划分眼妆区域、唇妆区域、粉底妆区域，根据区域妆容特点优化网络，实现局部上妆的精准化，降低网络优化的时间。并发布一个新的百人量级的同人妆前妆后数据集，数据来源于真人化妆网站。

With the rapid growth of the economy and the improvement of living standards, people's need for makeup is increasing, which has prompted the beauty industry to flourish. However, there are thousands of makeup styles on the internet, and many kinds of cosmetic products in the world. It has become a research hotspot that how to quickly and accurately find personalized cosmetic products from the reference-makeup images. As the core technology of the virtual makeup system, the makeup transfer technology has also attracted more and more attention. Makeup transfer is a task which can transfer the reference-makeup to the before-makeup face, where makeup style is kept. The main difficulties of this task include: 1) the lack of datasets which are composed of before-makeup and after-makeup images; 2) the difference of facial structures, such as human face, eyebrow distance, and lip shape; 3) the simplicity of transferring makeup style. 
Therefore, considering the different transfer styles to the same person, we proposed a fast multi-way regional makeup transfer deep network (FMaT), combined with the semantic information of high-level features of reference-makeup to optimize the network. Considering the facial structural difference between people, we established the dual network of makeup applier and makeup remover, respectively. Based on this, a belle cycle generative adversarial network (BelleGAN) is proposed to keep the structural features of the original face. Considering the simplicity of makeup style, a method of dividing the makeup region, which is proposed according to the different makeup of eyeshadow, lipstick and foundation, improves speed of network optimization and the expression ability of makeup features. The above makeup transfer algorithms are used for alternative makeup generation. Compared to the previous works, our models exhibit the advantage of smaller storage space and faster generation speed. They can keep the structure of the original face and lead to balancer eyeshadow, more vivid lipstick color and more detailed foundation make-up. We estimate our models on universal makeup transfer datasets. The experimental results show that these proposed methods achieve a better visual effect. Besides, they also overwhelm the alternatives in aspects of makeup speed, the effects of transferring different styles to the same person, and transferring the same style to the various people.
The main contributions of this dissertation can be summarized as follows:
1. A fast multi-way regional makeup transfer deep network (FMaT) is proposed. Specifically, we first detect the key points of faces and utilize these points to align different faces in an end-to-end style; then we use three way-specifical losses to optimize our makeup transfer network jointly. These losses are designed with the semantic information of high-level features of makeup, and make better makeup results; finally, we use poisson blending to fuse multi-way outputs.
2. A belle cycle generative adversarial network (BelleGAN) is proposed. The entire model only requires a makeup applied network and a makeup removal network to complete the mutual transformation between the before-makeup image domain and the reference-makeup image domain. The two networks, which interact with each other, are optimized by adversarial loss, cycle consistency loss and makeup loss. 
3. A method of dividing the makeup region is proposed, which divides the eyeshadow makeup area, the lipstick makeup area, and the foundation makeup area. By optimizing the network according to the characteristics of the regional makeup, we achieve the precision of localized makeup and improve speed of network optimization. Moreover, we publish a new dataset which is composed of 100 persons with the before-makeup and after-makeup images. These images gathered from the makeup websites.