Deep Reinforcement Learning for Query-Conditioned Video Summarization | |
Yujia Zhang1,2; Michael Kampffmeyer3; Xiaoguang Zhao1,2; Min Tan1,2 | |
发表期刊 | Applied Sciences - Basel |
2019-02 | |
卷号 | 9期号:4页码:750-765 |
摘要 | Query-conditioned video summarization requires to (1) find a diverse set of video shots/frames that are representative for the whole video, and that (2) the selected shots/frames are related to a given query. Thus it can be tailored to different user interests leading to a better personalized summary and differs from the generic video summarization which only focuses on video content. Our work targets this query-conditioned video summarization task, by first proposing a Mapping Network (MapNet) in order to express how related a shot is to a given query. MapNet helps establish the relation between the two different modalities (videos and query), which allows mapping of visual information to query space. After that, a deep reinforcement learning-based summarization network (SummNet) is developed to provide personalized summaries by integrating relatedness, representativeness and diversity rewards. These rewards jointly guide the agent to select the most representative and diversity video shots that are most related to the user query. Experimental results on a query-conditioned video summarization benchmark demonstrate the effectiveness of our proposed method, indicating the usefulness of the proposed mapping mechanism as well as the reinforcement learning approach. |
关键词 | Query-conditioned Video Summarization Deep Reinforcement Learning Visual-text Embedding Temporal Modeling Vision Application |
WOS记录号 | WOS:000460696500136 |
七大方向——子方向分类 | 多模态智能 |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/23598 |
专题 | 复杂系统管理与控制国家重点实验室_先进机器人 |
通讯作者 | Yujia Zhang |
作者单位 | 1.Institute of Automation, Chinese Academy of Sciences 2.University of Chinese Academy of Sciences 3.Machine Learning Group, UiT The Arctic University of Norway |
第一作者单位 | 中国科学院自动化研究所 |
通讯作者单位 | 中国科学院自动化研究所 |
推荐引用方式 GB/T 7714 | Yujia Zhang,Michael Kampffmeyer,Xiaoguang Zhao,et al. Deep Reinforcement Learning for Query-Conditioned Video Summarization[J]. Applied Sciences - Basel,2019,9(4):750-765. |
APA | Yujia Zhang,Michael Kampffmeyer,Xiaoguang Zhao,&Min Tan.(2019).Deep Reinforcement Learning for Query-Conditioned Video Summarization.Applied Sciences - Basel,9(4),750-765. |
MLA | Yujia Zhang,et al."Deep Reinforcement Learning for Query-Conditioned Video Summarization".Applied Sciences - Basel 9.4(2019):750-765. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
applsci-09-00750.pdf(3674KB) | 期刊论文 | 作者接受稿 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论