CASIA OpenIR  > 模式识别国家重点实验室  > 自然语言处理
Read, Watch, Listen and Summarize: Multi-modal Summarization for Asynchronous Text, Image, Audio and Video
Li, Haoran; Zhu, Junnan; Ma, Cong; Zhang, Jiajun; Zong, Chengqing
Source PublicationIEEE Transactions on Knowledge and Data Engineering
2018
Volume1Issue:1Pages:1
Abstract

Automatic text summarization is a fundamental NLP application that aims to condense a source text into a shorter version. The rapid increase in multimedia data transmission over the Internet necessitates multi-modal summarization (MMS) from asynchronous collections of text, image, audio and video. In this work, we propose an extractive MMS method that unites the techniques of NLP, speech processing and computer vision to explore the rich information contained in multi-modal data and to improve the quality of MMS. The key idea is to bridge the semantic gaps between multi-modal content. For audio information, we design an approach to selectively use its transcription and to infer the salience of the transcription with audio signals. For visual information, we learn the joint representations of text and images using a neural network. Then, we capture the coverage of the generated summary for important visual information through text-image matching or multi-modal topic modeling. Finally, all the multi-modal aspects are considered to generate a textual summary by maximizing the salience, non-redundancy, readability and coverage through the budgeted optimization of submodular functions. We further introduce a publicly available MMS corpus in English and Chinese. The experimental results demonstrate that our methods outperform other competitive baseline methods.

KeywordSummarization
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/23201
Collection模式识别国家重点实验室_自然语言处理
Corresponding AuthorZhang, Jiajun
Affiliation中国科学院自动化研究所
First Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Corresponding Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
Li, Haoran,Zhu, Junnan,Ma, Cong,et al. Read, Watch, Listen and Summarize: Multi-modal Summarization for Asynchronous Text, Image, Audio and Video[J]. IEEE Transactions on Knowledge and Data Engineering,2018,1(1):1.
APA Li, Haoran,Zhu, Junnan,Ma, Cong,Zhang, Jiajun,&Zong, Chengqing.(2018).Read, Watch, Listen and Summarize: Multi-modal Summarization for Asynchronous Text, Image, Audio and Video.IEEE Transactions on Knowledge and Data Engineering,1(1),1.
MLA Li, Haoran,et al."Read, Watch, Listen and Summarize: Multi-modal Summarization for Asynchronous Text, Image, Audio and Video".IEEE Transactions on Knowledge and Data Engineering 1.1(2018):1.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Li, Haoran]'s Articles
[Zhu, Junnan]'s Articles
[Ma, Cong]'s Articles
Baidu academic
Similar articles in Baidu academic
[Li, Haoran]'s Articles
[Zhu, Junnan]'s Articles
[Ma, Cong]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Li, Haoran]'s Articles
[Zhu, Junnan]'s Articles
[Ma, Cong]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.