PyMAF-X: Towards Well-Aligned Full-Body Model Regression From Monocular Images

doi:10.1109/TPAMI.2023.3271691

	PyMAF-X: Towards Well-Aligned Full-Body Model Regression From Monocular Images
	Zhang, Hongwen1 ; Tian, Yating 2; Zhang, Yuxiang 1; Li, Mengcheng 1; An, Liang 1; Sun, Zhenan3 ; Liu, Yebin 1
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
	2023-10-01
卷号	45 期号:10 页码:12287-12303
通讯作者	Liu, Yebin(liuyebin@mail.tsinghua.edu.cn)
摘要	We present PyMAF-X, a regression-based approach to recovering a parametric full-body model from a single image. This task is very challenging since minor parametric deviation may lead to noticeable misalignment between the estimated mesh and the input image. Moreover, when integrating part-specific estimations into the full-body model, existing solutions tend to either degrade the alignment or produce unnatural wrist poses. To address these issues, we propose a Pyramidal Mesh Alignment Feedback (PyMAF) loop in our regression network for well-aligned human mesh recovery and extend it as PyMAF-X for the recovery of expressive full-body models. The core idea of PyMAF is to leverage a feature pyramid and rectify the predicted parameters explicitly based on the mesh-image alignment status. Specifically, given the currently predicted parameters, mesh-aligned evidence will be extracted from finer-resolution features accordingly and fed back for parameter rectification. To enhance the alignment perception, an auxiliary dense supervision is employed to provide mesh-image correspondence guidance while spatial alignment attention is introduced to enable the awareness of the global contexts for our network. When extending PyMAF for full-body mesh recovery, an adaptive integration strategy is proposed in PyMAF-X to produce natural wrist poses while maintaining the well-aligned performance of the part-specific estimations. The efficacy of our approach is validated on several benchmark datasets for body, hand, face, and full-body mesh recovery, where PyMAF and PyMAF-X effectively improve the mesh-image alignment and achieve new The project page with code and video results can be found at https://www.liuyebin.com/pymaf-x.
关键词	Expressive human mesh recovery full-body motion capture mesh alignment feedback monocular 3D reconstruction
DOI	10.1109/TPAMI.2023.3271691
关键词[WOS]	HUMAN SHAPE ; POSE ; REPRESENTATION
收录类别	SCI
语种	英语
资助项目	National Key Ramp;D Program of China ; National Natural Science Foundation of China[2021ZD0113501] ; National Natural Science Foundation of China[62125107] ; National Natural Science Foundation of China[U1836217] ; China Postdoctoral Science Foundation[62276263] ; [2022M721844]
项目资助者	National Key Ramp;D Program of China ; National Natural Science Foundation of China ; China Postdoctoral Science Foundation
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:001068816800049
出版者	IEEE COMPUTER SOC
引用统计	被引频次：1[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/53048
专题	多模态人工智能系统全国重点实验室
通讯作者	Liu, Yebin
作者单位	1.Tsinghua Univ, Dept Automat, Beijing 100084, Peoples R China 2.Nanjing Univ, Dept Comp Sci & Technol, Nanjing 210023, Peoples R China 3.Chinese Acad Sci, Inst Automat, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Zhang, Hongwen,Tian, Yating,Zhang, Yuxiang,et al. PyMAF-X: Towards Well-Aligned Full-Body Model Regression From Monocular Images[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2023,45(10):12287-12303.
APA	Zhang, Hongwen.,Tian, Yating.,Zhang, Yuxiang.,Li, Mengcheng.,An, Liang.,...&Liu, Yebin.(2023).PyMAF-X: Towards Well-Aligned Full-Body Model Regression From Monocular Images.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,45(10),12287-12303.
MLA	Zhang, Hongwen,et al."PyMAF-X: Towards Well-Aligned Full-Body Model Regression From Monocular Images".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 45.10(2023):12287-12303.