Learning Semantic-Aware Spatial-Temporal Attention for Interpretable Action Recognition

doi:10.1109/TCSVT.2021.3137023

	Learning Semantic-Aware Spatial-Temporal Attention for Interpretable Action Recognition
	Fu, Jie 1,2; Gao, Junyu2,3 ; Xu, Changsheng2,4
发表期刊	IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
ISSN	1051-8215
	2022-08-01
卷号	32 期号:8 页码:5213-5224
通讯作者	Xu, Changsheng(csxu@nlpr.ia.ac.cn)
摘要	Human beings can concentrate on the most semantically relevant visual information when performing action recognition, so as to make reasonable and interpretable predictions. However, most existing approaches, which are applied to address visual tasks, neglect to explicitly imitate such ability for improving the performance and reliability of models. In this paper, we propose an interpretable action recognition framework that can not only improve the performance but also enhance the visual interpretability of 3D CNNs. Specifically, we design a semantic-aware attention module to learn correlative spatial-temporal attention for different action categories. To further leverage the rich semantics of features extracted from different layers, we design a hierarchical semantic fusion module with the help of the learned attention. The proposed two modules can enhance and complement each other, meanwhile, the semantic-aware attention module enjoys the plug-and-play merit. We evaluate our method on different benchmarks with comprehensive ablation studies and visualization analysis. Experimental results demonstrate the effectiveness of our method, showing favorable accuracy against state-of-the-arts while enhancing the semantic interpretability (Code will be available at this link https://github.com/PHDJieFu).
关键词	Visualization Semantics Task analysis Three-dimensional displays Feature extraction Solid modeling Predictive models Semantic-aware spatial-temporal attention interpretable action recognition
DOI	10.1109/TCSVT.2021.3137023
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Plan of China[2020AAA0106200] ; National Natural Science Foundation of China[62036012] ; National Natural Science Foundation of China[61721004] ; National Natural Science Foundation of China[62102415] ; National Natural Science Foundation of China[62072286] ; National Natural Science Foundation of China[61720106006] ; National Natural Science Foundation of China[61832002] ; National Natural Science Foundation of China[62072455] ; National Natural Science Foundation of China[62002355] ; National Natural Science Foundation of China[U1836220] ; National Natural Science Foundation of China[U1705262] ; Key Research Program of Frontier Sciences of the Chinese Academy of Sciences (CAS)[QYZDJSSW-JSC039] ; Beijing Natural Science Foundation[L201001]
项目资助者	National Key Research and Development Plan of China ; National Natural Science Foundation of China ; Key Research Program of Frontier Sciences of the Chinese Academy of Sciences (CAS) ; Beijing Natural Science Foundation
WOS研究方向	Engineering
WOS类目	Engineering, Electrical & Electronic
WOS记录号	WOS:000835828500026
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：10[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/49813
专题	多模态人工智能系统全国重点实验室_多媒体计算
通讯作者	Xu, Changsheng
作者单位	1.Zhengzhou Univ, Sch Informat Engn, Zhengzhou 450001, Peoples R China 2.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China 3.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China 4.Peng Cheng Lab, Shenzhen 518066, Peoples R China
第一作者单位	模式识别国家重点实验室
通讯作者单位	模式识别国家重点实验室
推荐引用方式 GB/T 7714	Fu, Jie,Gao, Junyu,Xu, Changsheng. Learning Semantic-Aware Spatial-Temporal Attention for Interpretable Action Recognition[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2022,32(8):5213-5224.
APA	Fu, Jie,Gao, Junyu,&Xu, Changsheng.(2022).Learning Semantic-Aware Spatial-Temporal Attention for Interpretable Action Recognition.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,32(8),5213-5224.
MLA	Fu, Jie,et al."Learning Semantic-Aware Spatial-Temporal Attention for Interpretable Action Recognition".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 32.8(2022):5213-5224.