CASIA OpenIR  > 复杂系统管理与控制国家重点实验室  > 先进机器人
Deep Reinforcement Learning for Query-Conditioned Video Summarization
Yujia Zhang1,2; Michael Kampffmeyer3; Xiaoguang Zhao1,2; Min Tan1,2
Source PublicationApplied Sciences - Basel
2019-02
Volume9Issue:4Pages:750-765
Abstract

Query-conditioned video summarization requires to (1) find a diverse set of video shots/frames that are representative for the whole video, and that (2) the selected shots/frames are related to a given query. Thus it can be tailored to different user interests leading to a better personalized summary and differs from the generic video summarization which only focuses on video content. Our work targets this query-conditioned video summarization task, by first proposing a Mapping Network (MapNet) in order to express how related a shot is to a given query. MapNet helps establish the relation between the two different modalities (videos and query), which allows mapping of visual information to query space. After that, a deep reinforcement learning-based summarization network (SummNet) is developed to provide personalized summaries by integrating relatedness, representativeness and diversity rewards. These rewards jointly guide the agent to select the most representative and diversity video shots that are most related to the user query. Experimental results on a query-conditioned video summarization benchmark demonstrate the effectiveness of our proposed method, indicating the usefulness of the proposed mapping mechanism as well as the reinforcement learning approach.

KeywordQuery-conditioned Video Summarization Deep Reinforcement Learning Visual-text Embedding Temporal Modeling Vision Application
WOS IDWOS:000460696500136
Citation statistics
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/23598
Collection复杂系统管理与控制国家重点实验室_先进机器人
Corresponding AuthorYujia Zhang
Affiliation1.Institute of Automation, Chinese Academy of Sciences
2.University of Chinese Academy of Sciences
3.Machine Learning Group, UiT The Arctic University of Norway
First Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Corresponding Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
Yujia Zhang,Michael Kampffmeyer,Xiaoguang Zhao,et al. Deep Reinforcement Learning for Query-Conditioned Video Summarization[J]. Applied Sciences - Basel,2019,9(4):750-765.
APA Yujia Zhang,Michael Kampffmeyer,Xiaoguang Zhao,&Min Tan.(2019).Deep Reinforcement Learning for Query-Conditioned Video Summarization.Applied Sciences - Basel,9(4),750-765.
MLA Yujia Zhang,et al."Deep Reinforcement Learning for Query-Conditioned Video Summarization".Applied Sciences - Basel 9.4(2019):750-765.
Files in This Item: Download All
File Name/Size DocType Version Access License
applsci-09-00750.pdf(3674KB)期刊论文作者接受稿开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Yujia Zhang]'s Articles
[Michael Kampffmeyer]'s Articles
[Xiaoguang Zhao]'s Articles
Baidu academic
Similar articles in Baidu academic
[Yujia Zhang]'s Articles
[Michael Kampffmeyer]'s Articles
[Xiaoguang Zhao]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Yujia Zhang]'s Articles
[Michael Kampffmeyer]'s Articles
[Xiaoguang Zhao]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: applsci-09-00750.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.