Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection

doi:10.1109/TCSVT.2018.2799968

	Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection
	Liu, Jiaying 1; Li, Yanghao 1; Song, Sijie 1; Xing, Junliang2 ; Lan, Cuiling 3; Zeng, Wenjun 3
发表期刊	IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
ISSN	1051-8215
	2019-09-01
卷号	29 期号:9 页码:2667-2682
通讯作者	Liu, Jiaying(liujiaying@pku.edu.cn)
摘要	Online action detection is a brand new challenge and plays a critical role in visual surveillance analytics. It goes one step further than a conventional action recognition task, which recognizes human actions from well-segmented clips. Online action detection is desired to identify the action type and localize action positions on the fly from the untrimmed stream data. In this paper, we propose a multi-modality multi-task recurrent neural network, which incorporates both RGB and Skeleton networks. We design different temporal modeling networks to capture specific characteristics from various modalities. Then, a deep long short-term memory subnetwork is utilized effectively to capture the complex long-range temporal dynamics, naturally avoiding the conventional sliding window design and thus ensuring high computational efficiency. Constrained by a multi-task objective function in the training phase, this network achieves superior detection performance and is capable of automatically localizing the start and end points of actions more accurately. Furthermore, embedding subtask of regression provides the ability to forecast the action prior to its occurrence. We evaluate the proposed method and several other methods in action detection and forecasting on the online action detection data set and gaming action data set datasets. Experimental results demonstrate that our model achieves the state-of-the-art performance on both tasks.
关键词	Action detection recurrent neural network multi-modality joint classification-regression
DOI	10.1109/TCSVT.2018.2799968
关键词[WOS]	ACTION RECOGNITION ; ENSEMBLE ; MOTION
收录类别	SCI
语种	英语
资助项目	NVIDIA Corporation ; NVIDIA Corporation
项目资助者	NVIDIA Corporation
WOS研究方向	Engineering
WOS类目	Engineering, Electrical & Electronic
WOS记录号	WOS:000489738900012
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：34[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/26636
专题	复杂系统认知与决策实验室_智能系统与工程
通讯作者	Liu, Jiaying
作者单位	1.Peking Univ, Inst Comp Sci & Technol, Beijing 100080, Peoples R China 2.Chinese Acad Sci, Inst Automat, Beijing 100080, Peoples R China 3.Microsoft Res Asia, Beijing 100080, Peoples R China
推荐引用方式 GB/T 7714	Liu, Jiaying,Li, Yanghao,Song, Sijie,et al. Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2019,29(9):2667-2682.
APA	Liu, Jiaying,Li, Yanghao,Song, Sijie,Xing, Junliang,Lan, Cuiling,&Zeng, Wenjun.(2019).Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,29(9),2667-2682.
MLA	Liu, Jiaying,et al."Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 29.9(2019):2667-2682.