CASIA OpenIR  > 多模态人工智能系统全国重点实验室
Identifying the key frames: An attention-aware sampling method for action recognition
Dong, Wenkai1,2,4; Zhang, Zhaoxiang1,2,3,4; Song, Chunfeng1,2,4; Tan, Tieniu1,2,4
Source PublicationPATTERN RECOGNITION
ISSN0031-3203
2022-10-01
Volume130Pages:11
Corresponding AuthorZhang, Zhaoxiang(zhaoxiang.zhang@ia.ac.cn)
AbstractDeep learning based methods have achieved remarkable progress in action recognition. Existing works mainly focus on designing novel deep architectures to learn video representations for action recognition. Most existing methods treat sampled frames equally and average all the frame-level predictions to generate video-level predictions at the testing stage. However, within a video, discriminative actions may occur sparsely in a few frames whereas most other frames are irrelevant to the ground truth which may even lead to wrong results. As a result, we think that the strategy of selecting relevant frames would be a further important key to enhance the existing deep learning based action recognition. In this paper, we propose an attention-aware sampling method for action recognition, which aims to discard the irrelevant and misleading frames and preserve the most discriminative frames. We formulate the process of mining key frames from videos as a Markov decision process and train the attention agent through deep reinforcement learning without extra labels. The agent takes features and predictions from the baseline model as inputs and generates importance scores for all frames. Moreover, our approach is extensible, which can be applied to different existing deep learning based action recognition models. We achieve very competitive action recognition performance on two widely used action recognition datasets. (c) 2022 Elsevier Ltd. All rights reserved.
KeywordAction recognition Deep learning Reinforcement learning Pseudo labels
DOI10.1016/j.patcog.2022.108797
WOS KeywordNETWORK
Indexed BySCI
Language英语
Funding ProjectMajor Project for New Generation of AI[2018AAA0100400] ; National Natural Science Foundation of China[61836014] ; National Natural Science Foundation of China[U21B2042] ; National Natural Science Foundation of China[62006231] ; National Natural Science Foundation of China[62072457] ; National Youth Talent Support Program
Funding OrganizationMajor Project for New Generation of AI ; National Natural Science Foundation of China ; National Youth Talent Support Program
WOS Research AreaComputer Science ; Engineering
WOS SubjectComputer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS IDWOS:001027089400007
PublisherELSEVIER SCI LTD
Citation statistics
Cited Times:7[WOS]   [WOS Record]     [Related Records in WOS]
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/53670
Collection多模态人工智能系统全国重点实验室
Corresponding AuthorZhang, Zhaoxiang
Affiliation1.Chinese Acad Sci CASIA, Inst Automat, Beijing, Peoples R China
2.Univ Chinese Acad Sci UCAS, Beijing, Peoples R China
3.Chinese Acad Sci, Ctr Artificial Intelligence & Robot, HKISI, Beijing, Peoples R China
4.Natl Lab Pattern Recognit NLPR, Beijing, Peoples R China
First Author AffilicationInstitute of Automation, Chinese Academy of Sciences;  Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
Corresponding Author AffilicationInstitute of Automation, Chinese Academy of Sciences;  Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
Recommended Citation
GB/T 7714
Dong, Wenkai,Zhang, Zhaoxiang,Song, Chunfeng,et al. Identifying the key frames: An attention-aware sampling method for action recognition[J]. PATTERN RECOGNITION,2022,130:11.
APA Dong, Wenkai,Zhang, Zhaoxiang,Song, Chunfeng,&Tan, Tieniu.(2022).Identifying the key frames: An attention-aware sampling method for action recognition.PATTERN RECOGNITION,130,11.
MLA Dong, Wenkai,et al."Identifying the key frames: An attention-aware sampling method for action recognition".PATTERN RECOGNITION 130(2022):11.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Dong, Wenkai]'s Articles
[Zhang, Zhaoxiang]'s Articles
[Song, Chunfeng]'s Articles
Baidu academic
Similar articles in Baidu academic
[Dong, Wenkai]'s Articles
[Zhang, Zhaoxiang]'s Articles
[Song, Chunfeng]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Dong, Wenkai]'s Articles
[Zhang, Zhaoxiang]'s Articles
[Song, Chunfeng]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.