Knowledge Commons of Institute of Automation,CAS
Inverse Kinematics Embedded Network for Robust Patient Anatomy Avatar Reconstruction From Multimodal Data | |
Tongxi Zhou1,2; Mingcong Chen3,4; Guanglin Cao1,2; Jian, Hu2,4; Hongbin Liu2,4,5 | |
发表期刊 | IEEE Robotics and Automation Letters |
ISSN | 2377-3766 |
2024 | |
卷号 | 9期号:4页码:3395-3402 |
通讯作者 | Liu, Hongbin(liuhongbin@ia.ac.cn) |
摘要 | —Patient modelling has a wide range of applications in medicine and healthcare, such as clinical teaching, surgery navigation and automatic robotized scanning. While patients are typically covered or occluded in medical scenes, directly regressing human meshes from single RGB images is challenging. To this end, we design a deep learning-based patient anatomy reconstruction network from RGB-D images with three key modules: 1) the attention-based multimodal fusion module, 2) the analytical inverse kinematics module and 3) the anatomical layer module. In our pipeline, the color and depth modality are fully fused by the multimodal attention module to obtain a cover-insensitive feature map. The estimated 3D keypoints, learned from the fused feature, are further converted to patient model parameters through the embedded analytical inverse kinematics module. To capture more detailed patient structures, we also present a parametric anatomy avatar by extending the Skinned Multi-Person Linear Model (SMPL) with internal bone and artery models. Final meshes are driven by the predicted parameters via the anatomical layer module, generating digital twins of patients. Experimental results on the Simultaneously-Collected Multimodal Lying Pose Dataset demonstrate that our approach surpasses state-of-the-art human mesh recovery methods and shows robustness to occlusions. |
关键词 | Image reconstruction Kinematics Three-dimensional displays Image color analysis Biomedical imaging Avatars Solid modeling Gesture posture and facial expressions deep learning for visual perception modeling and simulating humans RGB-D perception |
DOI | 10.1109/LRA.2024.3366418 |
收录类别 | SCI |
语种 | 英语 |
资助项目 | InnoHK Program |
项目资助者 | InnoHK Program |
WOS研究方向 | Robotics |
WOS类目 | Robotics |
WOS记录号 | WOS:001177958600019 |
出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
是否为代表性论文 | 否 |
七大方向——子方向分类 | 计算机图形学与虚拟现实 |
国重实验室规划方向分类 | 其他 |
是否有论文关联数据集需要存交 | 否 |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/57356 |
专题 | 多模态人工智能系统全国重点实验室_智能微创医疗技术团队 |
通讯作者 | Hongbin Liu |
作者单位 | 1.School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China 2.State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China 3.Department of Biomedical Engineering, City University of Hong Kong, Hong Kong 4.Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences, Hong Kong 5.School of Biomedical Engineering and Imaging Sciences, King’s College London, SE1 7EU London, U.K. |
第一作者单位 | 中国科学院自动化研究所 |
通讯作者单位 | 中国科学院自动化研究所 |
推荐引用方式 GB/T 7714 | Tongxi Zhou,Mingcong Chen,Guanglin Cao,et al. Inverse Kinematics Embedded Network for Robust Patient Anatomy Avatar Reconstruction From Multimodal Data[J]. IEEE Robotics and Automation Letters,2024,9(4):3395-3402. |
APA | Tongxi Zhou,Mingcong Chen,Guanglin Cao,Jian, Hu,&Hongbin Liu.(2024).Inverse Kinematics Embedded Network for Robust Patient Anatomy Avatar Reconstruction From Multimodal Data.IEEE Robotics and Automation Letters,9(4),3395-3402. |
MLA | Tongxi Zhou,et al."Inverse Kinematics Embedded Network for Robust Patient Anatomy Avatar Reconstruction From Multimodal Data".IEEE Robotics and Automation Letters 9.4(2024):3395-3402. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
Inverse_Kinematics_E(4590KB) | 期刊论文 | 作者接受稿 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论