Beyond 3DMM: Learning to Capture High-Fidelity 3D Face Shape

doi:10.1109/TPAMI.2022.3164131

	Beyond 3DMM: Learning to Capture High-Fidelity 3D Face Shape
	Zhu, Xiangyu1,2,3 ; Yu, Chang 1,2,3; Huang, Di 4; Lei, Zhen1,2,3,5 ; Wang, Hao 1,2,3; Li, Stan Z.6
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
	2023-02-01
卷号	45 期号:2 页码:1442-1457
通讯作者	Lei, Zhen(zlei@nlpr.ia.ac.cn)
摘要	3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori. However, previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry, which is attributed to insufficient ground-truth 3D shapes, unreliable training strategies and limited representation power of 3DMM. To alleviate this issue, this paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person. Specifically, given a 2D image as the input, we virtually render the image in several calibrated views to normalize pose variations while preserving the original image geometry. A many-to-one hourglass network serves as the encode-decoder to fuse multiview features and generate vertex displacements as the fine-grained geometry. Besides, the neural network is trained by directly optimizing the visual effect, where two 3D shapes are compared by measuring the similarity between the multiview images rendered from the shapes. Finally, we propose to generate the ground-truth 3D shapes by registering RGB-D images followed by pose and shape augmentation, providing sufficient data for network training. Experiments on several challenging protocols demonstrate the superior reconstruction accuracy of our proposal on the face shape.
关键词	3D face face reconstruction 3DMM fine-grained personalized 3D face dataset
DOI	10.1109/TPAMI.2022.3164131
关键词[WOS]	RECONSTRUCTION
收录类别	SCI
语种	英语
资助项目	National Key Research & Development Program[2020AAA0140002] ; Chinese National Natural Science Foundation[62176256] ; Chinese National Natural Science Foundation[61876178] ; Chinese National Natural Science Foundation[61872367] ; Chinese National Natural Science Foundation[61976229] ; Chinese National Natural Science Foundation[62106264] ; Chinese National Natural Science Foundation[62176012] ; Chinese National Natural Science Foundation[62022011] ; Youth Innovation Promotion Association CAS[Y2021131] ; InnoHK program
项目资助者	National Key Research & Development Program ; Chinese National Natural Science Foundation ; Youth Innovation Promotion Association CAS ; InnoHK program
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000912386000007
出版者	IEEE COMPUTER SOC
引用统计	被引频次：1[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/51339
专题	多模态人工智能系统全国重点实验室
通讯作者	Lei, Zhen
作者单位	1.Chinese Acad Sci, Ctr Biometr & Secur Res, Beijing 100190, Peoples R China 2.Chinese Acad Sci, Natl Lab Pattern Recognit, Inst Automat, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China 4.Beihang Univ, Key Lab Software Dev Environm, Sch Comp Sci & Engn, Beijing 100191, Peoples R China 5.Chinese Acad Sci, Ctr Artificial Intelligence & Robot, Hong Kong Inst Sci & Innovat, Hong Kong, Peoples R China 6.Westlake Univ, Sch Engn, Hangzhou 310024, Zhejiang, Peoples R China
第一作者单位	模式识别国家重点实验室
通讯作者单位	模式识别国家重点实验室
推荐引用方式 GB/T 7714	Zhu, Xiangyu,Yu, Chang,Huang, Di,et al. Beyond 3DMM: Learning to Capture High-Fidelity 3D Face Shape[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2023,45(2):1442-1457.
APA	Zhu, Xiangyu,Yu, Chang,Huang, Di,Lei, Zhen,Wang, Hao,&Li, Stan Z..(2023).Beyond 3DMM: Learning to Capture High-Fidelity 3D Face Shape.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,45(2),1442-1457.
MLA	Zhu, Xiangyu,et al."Beyond 3DMM: Learning to Capture High-Fidelity 3D Face Shape".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 45.2(2023):1442-1457.