Knowledge Commons of Institute of Automation,CAS
Pose-Appearance Relational Modeling for Video Action Recognition | |
Cui, Mengmeng1; Wang, Wei1; Zhang, Kunbo1,2; Sun, Zhenan1,2; Wang, Liang1,2 | |
发表期刊 | IEEE TRANSACTIONS ON IMAGE PROCESSING |
ISSN | 1057-7149 |
2023 | |
卷号 | 32页码:295-308 |
通讯作者 | Wang, Wei(wangwei@nlpr.ia.ac.cn) |
摘要 | Recent studies of video action recognition can be classified into two categories: the appearance-based methods and the pose-based methods. The appearance-based methods generally cannot model temporal dynamics of large motion well by virtue of optical flow estimation, while the pose-based methods ignore the visual context information such as typical scenes and objects, which are also important cues for action understanding. In this paper, we tackle these problems by proposing a Pose-Appearance Relational Network (PARNet), which models the correlation between human pose and image appearance, and combines the benefits of these two modalities to improve the robustness towards unconstrained real-world videos. There are three network streams in our model, namely pose stream, appearance stream and relation stream. For the pose stream, a Temporal Multi-Pose RNN module is constructed to obtain the dynamic representations through temporal modeling of 2D poses. For the appearance stream, a Spatial Appearance CNN module is employed to extract the global appearance representation of the video sequence. For the relation stream, a Pose-Aware RNN module is built to connect pose and appearance streams by modeling action-sensitive visual context information. Through jointly optimizing the three modules, PARNet achieves superior performances compared with the state-of-the-arts on both the pose-complete datasets (KTH, Penn-Action, UCF11) and the challenging pose-incomplete datasets (UCF101, HMDB51, JHMDB), demonstrating its robustness towards complex environments and noisy skeletons. Its effectiveness on NTU-RGBD dataset is also validated even compared with 3D skeleton-based methods. Furthermore, an appearance-enhanced PARNet equipped with a RGB-based I3D stream is proposed, which outperforms the Kinetics pre-trained competitors on UCF101 and HMDB51. The better experimental results verify the potentials of our framework by integrating various modules. |
关键词 | Action recognition 2D pose-appearance relational modeling temporal attention LSTM |
DOI | 10.1109/TIP.2022.3228156 |
关键词[WOS] | ATTENTION NETWORK ; LSTM |
收录类别 | SCI |
语种 | 英语 |
资助项目 | National Natural Science Foundation of China[61976214] ; National Natural Science Foundation of China[62071468] ; National Natural Science Foundation of China[62006225] ; National Natural Science Foundation of China[61806197] ; National Natural Science Foundation of China[6207146] ; Strategic Priority Research Program of Chinese Academy of Sciences[XDA27040700] ; Beijing Municipal Natural Science Foundation[4214075] |
项目资助者 | National Natural Science Foundation of China ; Strategic Priority Research Program of Chinese Academy of Sciences ; Beijing Municipal Natural Science Foundation |
WOS研究方向 | Computer Science ; Engineering |
WOS类目 | Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic |
WOS记录号 | WOS:000902111900021 |
出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/51085 |
专题 | 多模态人工智能系统全国重点实验室 |
通讯作者 | Wang, Wei |
作者单位 | 1.Chinese Acad Sci, Inst Automat, Ctr Res Intelligent Percept & Comp, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 101408, Peoples R China |
第一作者单位 | 模式识别国家重点实验室 |
通讯作者单位 | 模式识别国家重点实验室 |
推荐引用方式 GB/T 7714 | Cui, Mengmeng,Wang, Wei,Zhang, Kunbo,et al. Pose-Appearance Relational Modeling for Video Action Recognition[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2023,32:295-308. |
APA | Cui, Mengmeng,Wang, Wei,Zhang, Kunbo,Sun, Zhenan,&Wang, Liang.(2023).Pose-Appearance Relational Modeling for Video Action Recognition.IEEE TRANSACTIONS ON IMAGE PROCESSING,32,295-308. |
MLA | Cui, Mengmeng,et al."Pose-Appearance Relational Modeling for Video Action Recognition".IEEE TRANSACTIONS ON IMAGE PROCESSING 32(2023):295-308. |
条目包含的文件 | 条目无相关文件。 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论