CASIA OpenIR
Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection
Liu, Jiaying1; Li, Yanghao1; Song, Sijie1; Xing, Junliang2; Lan, Cuiling3; Zeng, Wenjun3
Source PublicationIEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
ISSN1051-8215
2019-09-01
Volume29Issue:9Pages:2667-2682
Corresponding AuthorLiu, Jiaying(liujiaying@pku.edu.cn)
AbstractOnline action detection is a brand new challenge and plays a critical role in visual surveillance analytics. It goes one step further than a conventional action recognition task, which recognizes human actions from well-segmented clips. Online action detection is desired to identify the action type and localize action positions on the fly from the untrimmed stream data. In this paper, we propose a multi-modality multi-task recurrent neural network, which incorporates both RGB and Skeleton networks. We design different temporal modeling networks to capture specific characteristics from various modalities. Then, a deep long short-term memory subnetwork is utilized effectively to capture the complex long-range temporal dynamics, naturally avoiding the conventional sliding window design and thus ensuring high computational efficiency. Constrained by a multi-task objective function in the training phase, this network achieves superior detection performance and is capable of automatically localizing the start and end points of actions more accurately. Furthermore, embedding subtask of regression provides the ability to forecast the action prior to its occurrence. We evaluate the proposed method and several other methods in action detection and forecasting on the online action detection data set and gaming action data set datasets. Experimental results demonstrate that our model achieves the state-of-the-art performance on both tasks.
KeywordAction detection recurrent neural network multi-modality joint classification-regression
DOI10.1109/TCSVT.2018.2799968
WOS KeywordACTION RECOGNITION ; ENSEMBLE ; MOTION
Indexed BySCI
Language英语
Funding ProjectNVIDIA Corporation ; NVIDIA Corporation
Funding OrganizationNVIDIA Corporation
WOS Research AreaEngineering
WOS SubjectEngineering, Electrical & Electronic
WOS IDWOS:000489738900012
PublisherIEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
Citation statistics
Cited Times:4[WOS]   [WOS Record]     [Related Records in WOS]
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/26636
Collection中国科学院自动化研究所
Corresponding AuthorLiu, Jiaying
Affiliation1.Peking Univ, Inst Comp Sci & Technol, Beijing 100080, Peoples R China
2.Chinese Acad Sci, Inst Automat, Beijing 100080, Peoples R China
3.Microsoft Res Asia, Beijing 100080, Peoples R China
Recommended Citation
GB/T 7714
Liu, Jiaying,Li, Yanghao,Song, Sijie,et al. Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2019,29(9):2667-2682.
APA Liu, Jiaying,Li, Yanghao,Song, Sijie,Xing, Junliang,Lan, Cuiling,&Zeng, Wenjun.(2019).Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,29(9),2667-2682.
MLA Liu, Jiaying,et al."Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 29.9(2019):2667-2682.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Liu, Jiaying]'s Articles
[Li, Yanghao]'s Articles
[Song, Sijie]'s Articles
Baidu academic
Similar articles in Baidu academic
[Liu, Jiaying]'s Articles
[Li, Yanghao]'s Articles
[Song, Sijie]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Liu, Jiaying]'s Articles
[Li, Yanghao]'s Articles
[Song, Sijie]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.