A Unified Framework for High Fidelity Face Swap and Expression Reenactment | |
Peng, Bo1,2; Fan, Hongxing1; Wang, Wei1; Dong, Jing1; Lyu, Siwei3 | |
发表期刊 | IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY |
ISSN | 1051-8215 |
2022-06-01 | |
卷号 | 32期号:6页码:3673-3684 |
通讯作者 | Dong, Jing(jdong@nlpr.ia.ac.cn) |
摘要 | Face manipulation techniques improve fast with the development of powerful image generation models. Two particular face manipulation methods, namely face swap and expression reenactment attract much attention for their flexibility and ease to generate high quality synthesis results. Recently, these two subjects are actively studied. However, most existing methods treat the two tasks separately, ignoring their underlying similarity. In this paper, we propose to tackle the two problems within a unified framework that achieves high quality synthesis results. The enabling component for our unified framework is the clean disentanglement of 3D pose, shape, and expression factors and then recombining them for different tasks accordingly. We then use the same set of 2D representations for face swap and expression reenactment tasks that are input to a common image translation model to directly generate the final synthetic images. Once trained, the proposed model can accomplish both face swap and expression reenactment tasks for previously unseen subjects. Comprehensive experiments and comparisons show that the proposed method achieves high fidelity results in multiple aspects, and it is especially good at faithfully preserving source facial shape in the face swap task, and accurately transferring facial movements in the expression reenactment task. |
关键词 | Faces Task analysis Videos Shape Three-dimensional displays Face recognition Information integrity Face swap expression reenactment 3DMM video manipulation |
DOI | 10.1109/TCSVT.2021.3106047 |
收录类别 | SCI |
语种 | 英语 |
资助项目 | National Natural Science Foundation of China (NSFC)[61902400] ; National Natural Science Foundation of China (NSFC)[U19B2038] ; National Natural Science Foundation of China (NSFC)[61972395] ; National Natural Science Foundation of China (NSFC)[61772529] ; National Natural Science Foundation of China (NSFC)[62076240] ; Beijing Natural Science Foundation[4192058] |
项目资助者 | National Natural Science Foundation of China (NSFC) ; Beijing Natural Science Foundation |
WOS研究方向 | Engineering |
WOS类目 | Engineering, Electrical & Electronic |
WOS记录号 | WOS:000805833400031 |
出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/49585 |
专题 | 智能感知与计算研究中心 |
通讯作者 | Dong, Jing |
作者单位 | 1.Chinese Acad Sci, Inst Automat, Ctr Res Intelligent Percept & Comp CRIPAC, Natl Lab Pattern Recognit NLPR, Beijing 100190, Peoples R China 2.Chinese Acad Sci, Inst Informat Engn, State Key Lab Informat Secur, Beijing 100093, Peoples R China 3.Univ Buffalo State Univ New York Buffalo, Sch Engn & Appl Sci, Buffalo, NY 14260 USA |
第一作者单位 | 模式识别国家重点实验室 |
通讯作者单位 | 模式识别国家重点实验室 |
推荐引用方式 GB/T 7714 | Peng, Bo,Fan, Hongxing,Wang, Wei,et al. A Unified Framework for High Fidelity Face Swap and Expression Reenactment[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2022,32(6):3673-3684. |
APA | Peng, Bo,Fan, Hongxing,Wang, Wei,Dong, Jing,&Lyu, Siwei.(2022).A Unified Framework for High Fidelity Face Swap and Expression Reenactment.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,32(6),3673-3684. |
MLA | Peng, Bo,et al."A Unified Framework for High Fidelity Face Swap and Expression Reenactment".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 32.6(2022):3673-3684. |
条目包含的文件 | 条目无相关文件。 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论