View Adaptive Neural Networks for High Performance Skeleton-Based Human Action Recognition

doi:10.1109/TPAMI.2019.2896631

	View Adaptive Neural Networks for High Performance Skeleton-Based Human Action Recognition
	Zhang, Pengfei 1; Lan, Cuiling 3; Xing, Junliang5 ; Zeng, Wenjun 4; Xue, Jianru 2; Zheng, Nanning 2
发表期刊	IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE
ISSN	0162-8828
	2019-08-01
卷号	41 期号:8 页码:1963-1978
通讯作者	Lan, Cuiling(culan@microsoft.com) ; Xue, Jianru(jrxue@mail.xjtu.edu.cn)
摘要	Skeleton-based human action recognition has recently attracted increasing attention thanks to the accessibility and the popularity of 3D skeleton data. One of the key challenges in action recognition lies in the large variations of action representations when they are captured from different viewpoints. In order to alleviate the effects of view variations, this paper introduces a novel view adaptation scheme, which automatically determines the virtual observation viewpoints over the course of an action in a learning based data driven manner. Instead of re-positioning the skeletons using a fixed human-defined prior criterion, we design two view adaptive neural networks, i.e., VA-RNN and VA-CNN, which are respectively built based on the recurrent neural network (RNN) with the Long Short-term Memory (LSTM) and the convolutional neural network (CNN). For each network, a novel view adaptation module learns and determines the most suitable observation viewpoints, and transforms the skeletons to those viewpoints for the end-to-end recognition with a main classification network. Ablation studies find that the proposed view adaptive models are capable of transforming the skeletons of various views to much more consistent virtual viewpoints. Therefore, the models largely eliminate the influence of the viewpoints, enabling the networks to focus on the learning of action-specific features and thus resulting in superior performance. In addition, we design a two-stream scheme (referred to as VA-fusion) that fuses the scores of the two networks to provide the final prediction, obtaining enhanced performance. Moreover, random rotation of skeleton sequences is employed to improve the robustness of view adaptation models and alleviate overfitting during training. Extensive experimental evaluations on five challenging benchmarks demonstrate the effectiveness of the proposed view-adaptive networks and superior performance over state-of-the-art approaches.
关键词	View adaptation skeleton action recognition RNN CNN consistent
DOI	10.1109/TPAMI.2019.2896631
关键词[WOS]	CLASSIFICATION ; MOTION ; ROBUST
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Program of China[2016YFB1001004] ; Natural Science Foundation of China[61672519] ; National Key Research and Development Program of China[2016YFB1001004] ; Natural Science Foundation of China[61672519]
项目资助者	National Key Research and Development Program of China ; Natural Science Foundation of China
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000473598800013
出版者	IEEE COMPUTER SOC
引用统计	被引频次：248[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/26841
专题	复杂系统认知与决策实验室_智能系统与工程
通讯作者	Lan, Cuiling; Xue, Jianru
作者单位	1.Xi An Jiao Tong Univ, Xian 710049, Shaanxi, Peoples R China 2.Xi An Jiao Tong Univ, Inst Artificial Intelligence & Robot, Xian 710049, Shaanxi, Peoples R China 3.Microsoft Res Asia, Beijing 100080, Peoples R China 4.Microsoft Res Asia, Senior Leadership Team, Beijing 100080, Peoples R China 5.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing, Peoples R China
推荐引用方式 GB/T 7714	Zhang, Pengfei,Lan, Cuiling,Xing, Junliang,et al. View Adaptive Neural Networks for High Performance Skeleton-Based Human Action Recognition[J]. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,2019,41(8):1963-1978.
APA	Zhang, Pengfei,Lan, Cuiling,Xing, Junliang,Zeng, Wenjun,Xue, Jianru,&Zheng, Nanning.(2019).View Adaptive Neural Networks for High Performance Skeleton-Based Human Action Recognition.IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE,41(8),1963-1978.
MLA	Zhang, Pengfei,et al."View Adaptive Neural Networks for High Performance Skeleton-Based Human Action Recognition".IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 41.8(2019):1963-1978.