CASIA OpenIR  > 模式识别实验室
Learning Dense Correspondence for NeRF-Based Face Reenactment
Songlin Yang1,2; Wei Wang2; Yushi Lan3; Xiangyu Fan4; Bo Peng2; Lei Yang4; Jing Dong2
2024
Conference NameThe 38th AAAI Conference on Artificial Intelligence
Conference Date2024年2月20日至27日
Conference Place加拿大渥太华
Publication Placett
Publishertt
Abstract

Face reenactment is challenging due to the need to establish
dense correspondence between various face representations
for motion transfer. Recent studies have utilized Neural Radi-
ance Field (NeRF) as fundamental representation, which fur-
ther enhanced the performance of multi-view face reenact-
ment in photo-realism and 3D consistency. However, estab-
lishing dense correspondence between different face NeRFs
is non-trivial, because implicit representations lack ground-
truth correspondence annotations like mesh-based 3D para-
metric models (e.g., 3DMM) with index-aligned vertexes. Al-
though aligning 3DMM space with NeRF-based face repre-
sentations can realize motion control, it is sub-optimal for
their limited face-only modeling and low identity fidelity.
Therefore, we are inspired to ask: Can we learn the dense
correspondence between different NeRF-based face repre-
sentations without a 3D parametric model prior? To ad-
dress this challenge, we propose a novel framework, which
adopts tri-planes as fundamental NeRF representation and
decomposes face tri-planes into three components: canoni-
cal tri-planes, identity deformations, and motion. In terms
of motion control, our key contribution is proposing a Plane
Dictionary (PlaneDict) module, which efficiently maps the
motion conditions to a linear weighted addition of learnable
orthogonal plane bases. To the best of our knowledge, our
framework is the first method that achieves one-shot multi-
view face reenactment without a 3D parametric model prior.
Extensive experiments demonstrate that we produce better
results in fine-grained motion control and identity preser-
vation than previous methods. Project page (video demo):
https://songlin1998.github.io/planedict/.

Keywordtt
MOST Discipline Cataloguett
DOItt
Indexed ByEI
Sub direction classification多模态智能
planning direction of the national heavy laboratory可解释人工智能
Paper associated data
Citation statistics
Document Type会议论文
Identifierhttp://ir.ia.ac.cn/handle/173211/57514
Collection模式识别实验室
Corresponding AuthorWei Wang
Affiliation1.School of Artificial Intelligence, University of Chinese Academy of Sciences, China
2.CRIPAC & MAIS, Institute of Automation, Chinese Academy of Sciences, China
3.S-Lab, Nanyang Technological University, Singapore
4.SenseTime, China
First Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Corresponding Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
Songlin Yang,Wei Wang,Yushi Lan,et al. Learning Dense Correspondence for NeRF-Based Face Reenactment[C]. tt:tt,2024.
Files in This Item: Download All
File Name/Size DocType Version Access License
AAAI_2024_multi_view(2179KB)会议论文 开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Songlin Yang]'s Articles
[Wei Wang]'s Articles
[Yushi Lan]'s Articles
Baidu academic
Similar articles in Baidu academic
[Songlin Yang]'s Articles
[Wei Wang]'s Articles
[Yushi Lan]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Songlin Yang]'s Articles
[Wei Wang]'s Articles
[Yushi Lan]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: AAAI_2024_multi_view_face_reenactment__Camera_Ready_.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.