Learning Proposal-Aware Re-Ranking for Weakly-Supervised Temporal Action Localization

doi:10.1109/TCSVT.2023.3283430

	Learning Proposal-Aware Re-Ranking for Weakly-Supervised Temporal Action Localization
	Hu, Yufan 1,2; Fu, Jie 3; Chen, Mengyuan 3; Gao, Junyu3 ; Dong, Jianfeng 4; Fan, Bin1,2 ; Liu, Hongmin 1,2
发表期刊	IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
ISSN	1051-8215
	2024
卷号	34 期号:1 页码:207-220
通讯作者	Liu, Hongmin(hmliu_82@163.com)
摘要	Weakly-supervised temporal action localization (WTAL) aims to localize and classify action instances in untrimmed videos with only video-level labels available. Despite the remarkable success of existing methods, whose generated proposals are commonly far more than the ground-truth action instances, it still makes sense to improve the ranking accuracy of the generated proposals since users in real-world scenarios usually prioritize the action proposals with the highest confidence scores. The inaccuracy of the proposal ranking mainly comes from two aspects: For one thing, the traditional proposal generation manner entirely relies on snippet-level perception, resulting in a significant yet unnoticed gap with the target of proposal-level localization. For another, existing methods commonly employ a hand-crafted proposal generation manner, a post-process that does not participate in model optimization. To address the above issues, we propose an end-to-end trained two-stage method, termed as Learning Proposal-aware Re-ranking (LPR) for WTAL. In the first stage, we design a proposal-aware feature learning module to inject the proposal-aware contextual information into each snippet, and then the enhanced features are utilized for predicting initial proposals. Furthermore, to perform effective and efficient proposal re-ranking, in the second stage, we contrast the proposals attached with high confidence scores with our constructed multi-scale foreground/background prototypes for further optimization. Evaluated by both the vanilla and Top- $k$ mAP metrics, results of extensive experiments on two popular benchmarks demonstrate the effectiveness of our proposed method.
关键词	Proposals Feature extraction Location awareness Videos Measurement Task analysis Optimization weakly-supervised temporal action localization Proposal-aware reranking
DOI	10.1109/TCSVT.2023.3283430
关键词[WOS]	NETWORK ; VIDEO ; RETRIEVAL ; ATTENTION
收录类别	SCI
语种	英语
资助项目	Beijing Natural Science Foundation
项目资助者	Beijing Natural Science Foundation
WOS研究方向	Engineering
WOS类目	Engineering, Electrical & Electronic
WOS记录号	WOS:001138814400041
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/55513
专题	多模态人工智能系统全国重点实验室
通讯作者	Liu, Hongmin
作者单位	1.Univ Sci & Technol Beijing, Key Lab Intelligent Bion Unmanned Syst, Minist Educ, Sch Intelligence Sci & Technol, Beijing 100083, Peoples R China 2.Univ Sci & Technol Beijing, Inst Artificial Intelligence, Beijing 100083, Peoples R China 3.Chinese Acad Sci, Inst Automat, State Key Lab Multimodal Artificial Intelligence S, Beijing 100190, Peoples R China 4.Zhejiang Gongshang Univ, Coll Comp & Informat Engn, Hangzhou 310018, Peoples R China
推荐引用方式 GB/T 7714	Hu, Yufan,Fu, Jie,Chen, Mengyuan,et al. Learning Proposal-Aware Re-Ranking for Weakly-Supervised Temporal Action Localization[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2024,34(1):207-220.
APA	Hu, Yufan.,Fu, Jie.,Chen, Mengyuan.,Gao, Junyu.,Dong, Jianfeng.,...&Liu, Hongmin.(2024).Learning Proposal-Aware Re-Ranking for Weakly-Supervised Temporal Action Localization.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,34(1),207-220.
MLA	Hu, Yufan,et al."Learning Proposal-Aware Re-Ranking for Weakly-Supervised Temporal Action Localization".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 34.1(2024):207-220.