Inverse Kinematics Embedded Network for Robust Patient Anatomy Avatar Reconstruction From Multimodal Data
Tongxi Zhou1,2; Mingcong Chen3,4; Guanglin Cao1,2; Jian, Hu2,4; Hongbin Liu2,4,5
Source PublicationIEEE Robotics and Automation Letters
ISSN2377-3766
2024
Volume9Issue:4Pages:3395-3402
Corresponding AuthorLiu, Hongbin(liuhongbin@ia.ac.cn)
Abstract

—Patient modelling has a wide range of applications in medicine and healthcare, such as clinical teaching, surgery navigation and automatic robotized scanning. While patients are typically covered or occluded in medical scenes, directly regressing human meshes from single RGB images is challenging. To this end, we design a deep learning-based patient anatomy reconstruction network from RGB-D images with three key modules: 1) the attention-based multimodal fusion module, 2) the analytical inverse kinematics module and 3) the anatomical layer module. In our pipeline, the color and depth modality are fully fused by the multimodal attention module to obtain a cover-insensitive feature map. The estimated 3D keypoints, learned from the fused feature, are further converted to patient model parameters through the embedded analytical inverse kinematics module. To capture more detailed patient structures, we also present a parametric anatomy avatar by extending the Skinned Multi-Person Linear Model (SMPL) with internal bone and artery models. Final meshes are driven by the predicted parameters via the anatomical layer module, generating digital twins of patients. Experimental results on the Simultaneously-Collected Multimodal Lying Pose Dataset demonstrate that our approach surpasses state-of-the-art human mesh recovery methods and shows robustness to occlusions.

KeywordImage reconstruction Kinematics Three-dimensional displays Image color analysis Biomedical imaging Avatars Solid modeling Gesture posture and facial expressions deep learning for visual perception modeling and simulating humans RGB-D perception
DOI10.1109/LRA.2024.3366418
Indexed BySCI
Language英语
Funding ProjectInnoHK Program
Funding OrganizationInnoHK Program
WOS Research AreaRobotics
WOS SubjectRobotics
WOS IDWOS:001177958600019
PublisherIEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
IS Representative Paper
Sub direction classification计算机图形学与虚拟现实
planning direction of the national heavy laboratory其他
Paper associated data
Citation statistics
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/57356
Collection多模态人工智能系统全国重点实验室_智能微创医疗技术团队
Corresponding AuthorHongbin Liu
Affiliation1.School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
2.State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
3.Department of Biomedical Engineering, City University of Hong Kong, Hong Kong
4.Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong
5.School of Biomedical Engineering and Imaging Sciences, King’s College London, SE1 7EU London, U.K.
First Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Corresponding Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
Tongxi Zhou,Mingcong Chen,Guanglin Cao,et al. Inverse Kinematics Embedded Network for Robust Patient Anatomy Avatar Reconstruction From Multimodal Data[J]. IEEE Robotics and Automation Letters,2024,9(4):3395-3402.
APA Tongxi Zhou,Mingcong Chen,Guanglin Cao,Jian, Hu,&Hongbin Liu.(2024).Inverse Kinematics Embedded Network for Robust Patient Anatomy Avatar Reconstruction From Multimodal Data.IEEE Robotics and Automation Letters,9(4),3395-3402.
MLA Tongxi Zhou,et al."Inverse Kinematics Embedded Network for Robust Patient Anatomy Avatar Reconstruction From Multimodal Data".IEEE Robotics and Automation Letters 9.4(2024):3395-3402.
Files in This Item: Download All
File Name/Size DocType Version Access License
Inverse_Kinematics_E(4590KB)期刊论文作者接受稿开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Tongxi Zhou]'s Articles
[Mingcong Chen]'s Articles
[Guanglin Cao]'s Articles
Baidu academic
Similar articles in Baidu academic
[Tongxi Zhou]'s Articles
[Mingcong Chen]'s Articles
[Guanglin Cao]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Tongxi Zhou]'s Articles
[Mingcong Chen]'s Articles
[Guanglin Cao]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: Inverse_Kinematics_Embedded_Network_for_Robust_Patient_Anatomy_Avatar_Reconstruction_From_Multimodal_Data.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.