Knowledge Commons of Institute of Automation,CAS
Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection | |
Liu, Jiaying1; Li, Yanghao1; Song, Sijie1; Xing, Junliang2; Lan, Cuiling3; Zeng, Wenjun3 | |
发表期刊 | IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY |
ISSN | 1051-8215 |
2019-09-01 | |
卷号 | 29期号:9页码:2667-2682 |
通讯作者 | Liu, Jiaying(liujiaying@pku.edu.cn) |
摘要 | Online action detection is a brand new challenge and plays a critical role in visual surveillance analytics. It goes one step further than a conventional action recognition task, which recognizes human actions from well-segmented clips. Online action detection is desired to identify the action type and localize action positions on the fly from the untrimmed stream data. In this paper, we propose a multi-modality multi-task recurrent neural network, which incorporates both RGB and Skeleton networks. We design different temporal modeling networks to capture specific characteristics from various modalities. Then, a deep long short-term memory subnetwork is utilized effectively to capture the complex long-range temporal dynamics, naturally avoiding the conventional sliding window design and thus ensuring high computational efficiency. Constrained by a multi-task objective function in the training phase, this network achieves superior detection performance and is capable of automatically localizing the start and end points of actions more accurately. Furthermore, embedding subtask of regression provides the ability to forecast the action prior to its occurrence. We evaluate the proposed method and several other methods in action detection and forecasting on the online action detection data set and gaming action data set datasets. Experimental results demonstrate that our model achieves the state-of-the-art performance on both tasks. |
关键词 | Action detection recurrent neural network multi-modality joint classification-regression |
DOI | 10.1109/TCSVT.2018.2799968 |
关键词[WOS] | ACTION RECOGNITION ; ENSEMBLE ; MOTION |
收录类别 | SCI |
语种 | 英语 |
资助项目 | NVIDIA Corporation ; NVIDIA Corporation |
项目资助者 | NVIDIA Corporation |
WOS研究方向 | Engineering |
WOS类目 | Engineering, Electrical & Electronic |
WOS记录号 | WOS:000489738900012 |
出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/26636 |
专题 | 复杂系统认知与决策实验室_智能系统与工程 |
通讯作者 | Liu, Jiaying |
作者单位 | 1.Peking Univ, Inst Comp Sci & Technol, Beijing 100080, Peoples R China 2.Chinese Acad Sci, Inst Automat, Beijing 100080, Peoples R China 3.Microsoft Res Asia, Beijing 100080, Peoples R China |
推荐引用方式 GB/T 7714 | Liu, Jiaying,Li, Yanghao,Song, Sijie,et al. Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2019,29(9):2667-2682. |
APA | Liu, Jiaying,Li, Yanghao,Song, Sijie,Xing, Junliang,Lan, Cuiling,&Zeng, Wenjun.(2019).Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,29(9),2667-2682. |
MLA | Liu, Jiaying,et al."Multi-Modality Multi-Task Recurrent Neural Network for Online Action Detection".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 29.9(2019):2667-2682. |
条目包含的文件 | 条目无相关文件。 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论