Emotion Knowledge Driven Video Highlight Detection

doi:10.1109/TMM.2020.3035285

	Emotion Knowledge Driven Video Highlight Detection
	Qi, Fan 1,2; Yang, Xiaoshan2,3,4 ; Xu, Changsheng2,3,4
发表期刊	IEEE TRANSACTIONS ON MULTIMEDIA
ISSN	1520-9210
	2021
卷号	23 页码:3999-4013
通讯作者	Xu, Changsheng(csxu@nlpr.ia.ac.cn)
摘要	This paper addresses video highlight detection which aims to select a small subset of frames according to user's major or special interest. The performances of conventional methods highly depend on large-scale manually labeled training data which are time-consuming and labor-intensive to collect. To deal with this problem, we trace back to the original problem definition and find that whether a user is interested in a specific video segment heavily depends on human's subjective emotions. Leveraging this insight, we introduce an emotion knowledge driven video detection framework for modeling human's general emotion and inferencing highlight strength. Firstly, we obtain the concept-level representation of the video clip with a front-end network. The concepts are used as nodes to build an emotion-related knowledge graph, and their relationships in the graph are modeled via external public knowledge graphs. Then we adopt Siamese GCNs to model the dependencies between nodes in the graph and propagate messages along the edges. Finally, we compute the emotion-aware representation of the video clip based on the GCN layers and further use it to predict the highlight score. Our framework, including the front-end network, graph convolution layers and the highlight mapping network, can be trained in an end-to-end manner with the constraint of a ranking loss. Experiments on two benchmark datasets show that our proposed method performs favorably against the state-of-the-art methods.
关键词	Visualization Training data Predictive models Training Semantics Emotion recognition Computational modeling Deep ranking knowledge graph video highlight detection
DOI	10.1109/TMM.2020.3035285
关键词[WOS]	RETRIEVAL
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Program of China[2018AAA0100604] ; National Natural Science Foundation of China[61720106006] ; National Natural Science Foundation of China[62072455] ; National Natural Science Foundation of China[61702511] ; National Natural Science Foundation of China[61751211] ; National Natural Science Foundation of China[61620106003] ; National Natural Science Foundation of China[61532009] ; National Natural Science Foundation of China[U1836220] ; National Natural Science Foundation of China[U1705262] ; National Natural Science Foundation of China[61872424] ; Key Research Program of Frontier Sciences of CAS[QYZDJSSWJSC039] ; Research Program of National Laboratory of Pattern Recognition[Z-2018007]
项目资助者	National Key Research and Development Program of China ; National Natural Science Foundation of China ; Key Research Program of Frontier Sciences of CAS ; Research Program of National Laboratory of Pattern Recognition
WOS研究方向	Computer Science ; Telecommunications
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS记录号	WOS:000720519900007
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
七大方向——子方向分类	图像视频处理与分析
引用统计	被引频次：11[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/46456
专题	多模态人工智能系统全国重点实验室_多媒体计算
通讯作者	Xu, Changsheng
作者单位	1.HeFei Univ Technol, Comp Sci, Hefei 230009, Peoples R China 2.Peng Cheng Lab, Shenzhen 518055, Peoples R China 3.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China 4.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100190, Peoples R China
通讯作者单位	模式识别国家重点实验室
推荐引用方式 GB/T 7714	Qi, Fan,Yang, Xiaoshan,Xu, Changsheng. Emotion Knowledge Driven Video Highlight Detection[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2021,23:3999-4013.
APA	Qi, Fan,Yang, Xiaoshan,&Xu, Changsheng.(2021).Emotion Knowledge Driven Video Highlight Detection.IEEE TRANSACTIONS ON MULTIMEDIA,23,3999-4013.
MLA	Qi, Fan,et al."Emotion Knowledge Driven Video Highlight Detection".IEEE TRANSACTIONS ON MULTIMEDIA 23(2021):3999-4013.