CASIA OpenIR
Gesture recognition based on deep deformable 3D convolutional neural networks
Zhang, Yifan1,2,3; Shi, Lei1,2,3; Wu, Yi4; Cheng, Ke1,2,3; Cheng, Jian1,2,3,5; Lu, Hanqing1,2,3
Source PublicationPATTERN RECOGNITION
ISSN0031-3203
2020-11-01
Issue107Pages:12
Abstract

Dynamic gesture recognition, which plays an essential role in human-computer interaction, has been widely investigated but not yet fully addressed. The challenge mainly lies in three folders: 1) to model both of the spatial appearance and the temporal evolution simultaneously; 2) to address the interference from the varied and complex background; 3) the requirement of real-time processing. In this paper, we address the above challenges by proposing a novel deep deformable 3D convolutional neural network for end-to-end learning, which not only gains impressive accuracy in challenging datasets but also can meet the requirement of the real-time processing. We propose three types of very deep 3D CNNs for gesture recognition, which can directly model the spatiotemporal information with their inherent hierarchical structure. To eliminate the background interference, a light-weight spatiotemporal deformable convolutional module is specially designed to augment the spatiotemporal sampling locations of the 3D convolution by learning additional offsets according to the preceding feature map. It can not only diversify the shape of the convolution kernel to better fit the appearance of the hands and arms, but also help the models pay more attention to the discriminative frames in the video sequence. The proposed method is evaluated on three challenging datasets, EgoGesture, Jester and Chalearn-IsoGD, and achieves the state-of-the-art performance on all of them. Our model ranked first on Jester's official leader-board until the submission time. The code and the trained models are released for better communication and future works(1). (C) 2020 Elsevier Ltd. All rights reserved.

KeywordGesture recognition Spatiotemporal deformable convolution Spatiotemporal convolutional neural network
DOI10.1016/j.patcog.2020.107416
WOS KeywordDATASET ; FUSION ; TIME
Indexed BySCI
Language英语
Funding ProjectNSFC[61876182] ; NSFC[61872364] ; NSFC[61876086] ; Jiangsu Frontier Technology Basic Research Project[BK20192004]
Funding OrganizationNSFC ; Jiangsu Frontier Technology Basic Research Project
WOS Research AreaComputer Science ; Engineering
WOS SubjectComputer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS IDWOS:000552866000006
PublisherELSEVIER SCI LTD
Citation statistics
Cited Times:1[WOS]   [WOS Record]     [Related Records in WOS]
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/40290
Collection中国科学院自动化研究所
Corresponding AuthorZhang, Yifan
Affiliation1.Chinese Acad Sci, Inst Automat, NLPR, Beijing, Peoples R China
2.Chinese Acad Sci, Inst Automat, AIRIA, Beijing, Peoples R China
3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing, Peoples R China
4.Wormpex AI Res, Bellevue, WA USA
5.CAS Ctr Excellence Brain Sci & Intelligence Techn, Beijing, Peoples R China
First Author AffilicationChinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China;  Institute of Automation, Chinese Academy of Sciences
Corresponding Author AffilicationChinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China;  Institute of Automation, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
Zhang, Yifan,Shi, Lei,Wu, Yi,et al. Gesture recognition based on deep deformable 3D convolutional neural networks[J]. PATTERN RECOGNITION,2020(107):12.
APA Zhang, Yifan,Shi, Lei,Wu, Yi,Cheng, Ke,Cheng, Jian,&Lu, Hanqing.(2020).Gesture recognition based on deep deformable 3D convolutional neural networks.PATTERN RECOGNITION(107),12.
MLA Zhang, Yifan,et al."Gesture recognition based on deep deformable 3D convolutional neural networks".PATTERN RECOGNITION .107(2020):12.
Files in This Item: Download All
File Name/Size DocType Version Access License
Online Version.pdf(1310KB)期刊论文作者接受稿开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Zhang, Yifan]'s Articles
[Shi, Lei]'s Articles
[Wu, Yi]'s Articles
Baidu academic
Similar articles in Baidu academic
[Zhang, Yifan]'s Articles
[Shi, Lei]'s Articles
[Wu, Yi]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Zhang, Yifan]'s Articles
[Shi, Lei]'s Articles
[Wu, Yi]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: Online Version.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.