Multi-Modality Self-Distillation for Weakly Supervised Temporal Action Localization

doi:10.1109/TIP.2021.3137649

CASIA OpenIR > 模式识别实验室

	Multi-Modality Self-Distillation for Weakly Supervised Temporal Action Localization
	Huang, Linjiang 1,2; Wang, Liang3 ; Li, Hongsheng 1,2
发表期刊	IEEE TRANSACTIONS ON IMAGE PROCESSING
ISSN	1057-7149
	2022
卷号	31 页码:1504-1519
通讯作者	Li, Hongsheng(hsli@ee.cuhk.edu.hk)
摘要	As a challenging task of high-level video understanding, Weakly-supervised Temporal Action Localization (WTAL) has attracted increasing attention in recent years. However, due to the weak supervisions of whole-video classification labels, it is challenging to accurately determine action instance boundaries. To address this issue, pseudo-label-based methods [Alwassel et al. (2019), Luo et al. (2020), and Zhai et al. (2020)] were proposed to generate snippet-level pseudo labels from classification results. In spite of the promising performance, these methods hardly take full advantages of multiple modalities, i.e., RGB and optical flow sequences, to generate high quality pseudo labels. Most of them ignored how to mitigate the label noise, which hinders the capability of the network on learning discriminative feature representations. To address these challenges, we propose a Multi-Modality Self-Distillation (MMSD) framework, which contains two single-modal streams and a fused-modal stream to perform multi-modality knowledge distillation and multi-modality self-voting. On the one hand, multi-modality knowledge distillation improves snippet-level classification performance by transferring knowledge between single-modal streams and a fused-modal stream. On the other hand, multi-modality self-voting mitigates the label noise in a modality voting manner according to the reliability and complementarity of the streams. Experimental results on THUMOS14 and ActivityNet1.3 datasets demonstrate the effectiveness of our method and superior performance over state-of-the-art approaches. Our code is available at https://github.com/LeonHLJ/MMSD.
关键词	Location awareness Reliability Noise measurement Annotations Training Head Task analysis Weakly supervised temporal action localization multi-modality pseudo label self-distillation
DOI	10.1109/TIP.2021.3137649
收录类别	SCI
语种	英语
资助项目	Centre for Perceptual and Interactive Intelligence Ltd. ; Research Grants Council of Hong Kong[14204021] ; Research Grants Council of Hong Kong[14208417] ; Research Grants Council of Hong Kong[14207319] ; Chinese University of Hong Kong (CUHK) Strategic Fund
项目资助者	Centre for Perceptual and Interactive Intelligence Ltd. ; Research Grants Council of Hong Kong ; Chinese University of Hong Kong (CUHK) Strategic Fund
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic
WOS记录号	WOS:000748370500006
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：12[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/47338
专题	模式识别实验室
通讯作者	Li, Hongsheng
作者单位	1.Ctr Perceptual & Interact Intelligence CPII, Hong Kong, Peoples R China 2.Chinese Univ Hong Kong, Multimedia Lab, Hong Kong, Peoples R China 3.Chinese Acad Sci CASIA, Inst Automat, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Huang, Linjiang,Wang, Liang,Li, Hongsheng. Multi-Modality Self-Distillation for Weakly Supervised Temporal Action Localization[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2022,31:1504-1519.
APA	Huang, Linjiang,Wang, Liang,&Li, Hongsheng.(2022).Multi-Modality Self-Distillation for Weakly Supervised Temporal Action Localization.IEEE TRANSACTIONS ON IMAGE PROCESSING,31,1504-1519.
MLA	Huang, Linjiang,et al."Multi-Modality Self-Distillation for Weakly Supervised Temporal Action Localization".IEEE TRANSACTIONS ON IMAGE PROCESSING 31(2022):1504-1519.