CASIA OpenIR  > 模式识别国家重点实验室  > 自然语言处理
Read, Watch, Listen and Summarize: Multi-modal Summarization for Asynchronous Text, Image, Audio and Video
Li, Haoran; Zhu, Junnan; Ma, Cong; Zhang, Jiajun; Zong, Chengqing
发表期刊IEEE Transactions on Knowledge and Data Engineering
2018
卷号1期号:1页码:1
摘要

Automatic text summarization is a fundamental NLP application that aims to condense a source text into a shorter version. The rapid increase in multimedia data transmission over the Internet necessitates multi-modal summarization (MMS) from asynchronous collections of text, image, audio and video. In this work, we propose an extractive MMS method that unites the techniques of NLP, speech processing and computer vision to explore the rich information contained in multi-modal data and to improve the quality of MMS. The key idea is to bridge the semantic gaps between multi-modal content. For audio information, we design an approach to selectively use its transcription and to infer the salience of the transcription with audio signals. For visual information, we learn the joint representations of text and images using a neural network. Then, we capture the coverage of the generated summary for important visual information through text-image matching or multi-modal topic modeling. Finally, all the multi-modal aspects are considered to generate a textual summary by maximizing the salience, non-redundancy, readability and coverage through the budgeted optimization of submodular functions. We further introduce a publicly available MMS corpus in English and Chinese. The experimental results demonstrate that our methods outperform other competitive baseline methods.

关键词Summarization
文献类型期刊论文
条目标识符http://ir.ia.ac.cn/handle/173211/23201
专题模式识别国家重点实验室_自然语言处理
通讯作者Zhang, Jiajun
作者单位中国科学院自动化研究所
第一作者单位中国科学院自动化研究所
通讯作者单位中国科学院自动化研究所
推荐引用方式
GB/T 7714
Li, Haoran,Zhu, Junnan,Ma, Cong,et al. Read, Watch, Listen and Summarize: Multi-modal Summarization for Asynchronous Text, Image, Audio and Video[J]. IEEE Transactions on Knowledge and Data Engineering,2018,1(1):1.
APA Li, Haoran,Zhu, Junnan,Ma, Cong,Zhang, Jiajun,&Zong, Chengqing.(2018).Read, Watch, Listen and Summarize: Multi-modal Summarization for Asynchronous Text, Image, Audio and Video.IEEE Transactions on Knowledge and Data Engineering,1(1),1.
MLA Li, Haoran,et al."Read, Watch, Listen and Summarize: Multi-modal Summarization for Asynchronous Text, Image, Audio and Video".IEEE Transactions on Knowledge and Data Engineering 1.1(2018):1.
条目包含的文件
条目无相关文件。
个性服务
推荐该条目
保存到收藏夹
查看访问统计
导出为Endnote文件
谷歌学术
谷歌学术中相似的文章
[Li, Haoran]的文章
[Zhu, Junnan]的文章
[Ma, Cong]的文章
百度学术
百度学术中相似的文章
[Li, Haoran]的文章
[Zhu, Junnan]的文章
[Ma, Cong]的文章
必应学术
必应学术中相似的文章
[Li, Haoran]的文章
[Zhu, Junnan]的文章
[Ma, Cong]的文章
相关权益政策
暂无数据
收藏/分享
所有评论 (0)
暂无评论
 

除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。