Knowledge Commons of Institute of Automation,CAS
DPE: Disentanglement of Pose and Expression for General Video Portrait Editing | |
Pang YX(庞有鑫)1,2![]() ![]() ![]() ![]() ![]() | |
2023-03 | |
会议名称 | IEEE Conference on Computer Vision and Pattern Recognition(CVPR) |
会议日期 | 2023-06 |
会议地点 | 加拿大 |
摘要 | One-shot video-driven talking face generation aims at producing a synthetic talking video by transferring the facial motion from a video to an arbitrary portrait image. Head pose and facial expression are always entangled in facial motion and transferred simultaneously. However, the entanglement sets up a barrier for these methods to be used in video portrait editing directly, where it may require to modify the expression only while maintaining the pose unchanged. One challenge of decoupling pose and expression is the lack of paired data, such as the same pose but different expressions. Only a few methods attempt to tackle this challenge with the feat of 3D Morphable Models (3DMMs) for explicit disentanglement. But 3DMMs are not accurate enough to capture facial details due to the limited number of Blendshapes, which has side effects on motion transfer. In this paper, we introduce a novel self-supervised disentanglement framework to decouple pose and expression without 3DMMs and paired data, which consists of a motion editing module, a pose generator, and an expression generator. The editing module projects faces into a latent space where pose motion and expression motion can be disentangled, and the pose or expression transfer can be performed in the latent space conveniently via addition. The two generators render the modified latent codes to images, respectively. Moreover, to guarantee the disentanglement, we propose a bidirectional cyclic training strategy with well-designed constraints. Evaluations demonstrate our method can control pose or expression independently and be used for general video editing. Code: https://github.com/Carlyx/DPE. |
七大方向——子方向分类 | 图像视频处理与分析 |
国重实验室规划方向分类 | 多模态协同认知 |
是否有论文关联数据集需要存交 | 否 |
文献类型 | 会议论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/52023 |
专题 | 多模态人工智能系统全国重点实验室 |
通讯作者 | Yan DM(严冬明) |
作者单位 | 1.自动化研究所 2.国科大人工智能学院 3.腾讯AI Lab |
推荐引用方式 GB/T 7714 | Pang YX,Zhang Y,Quan WZ,et al. DPE: Disentanglement of Pose and Expression for General Video Portrait Editing[C],2023. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
Pang_DPE_Disentangle(2486KB) | 会议论文 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论