CASIA OpenIR
Read, Watch, Listen, and Summarize: Multi-Modal Summarization for Asynchronous Text, Image, Audio and Video
Li, Haoran1,2; Zhu, Junnan1,2; Ma, Cong1,2; Zhang, Jiajun1,2; Zong, Chengqing3,4,5
Source PublicationIEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING
ISSN1041-4347
2019-05-01
Volume31Issue:5Pages:996-1009
Corresponding AuthorLi, Haoran(haoran.li@nlpr.ia.ac.cn)
AbstractAutomatic text summarization is a fundamental natural language processing (NLP) application that aims to condense a source text into a shorter version. The rapid increase in multimedia data transmission over the Internet necessitates multi-modal summarization (MMS) from asynchronous collections of text, image, audio, and video. In this work, we propose an extractive MMS method that unites the techniques of NLP, speech processing, and computer vision to explore the rich information contained in multi-modal data and to improve the quality of multimedia news summarization. The key idea is to bridge the semantic gaps between multi-modal content. Audio and visual are main modalities in the video. For audio information, we design an approach to selectively use its transcription and to infer the salience of the transcription with audio signals. For visual information, we learn the joint representations of text and images using a neural network. Then, we capture the coverage of the generated summary for important visual information through text-image matching or multi-modal topic modeling. Finally, all the multi-modal aspects are considered to generate a textual summary by maximizing the salience, non-redundancy, readability, and coverage through the budgeted optimization of submodular functions. We further introduce a publicly available MMS corpus in English and Chinese. 1 The experimental results obtained on our dataset demonstrate that our methods based on image matching and image topic framework outperform other competitive baseline methods.
KeywordSummarization multimedia multi-modal cross-modal natural language processing computer vision
DOI10.1109/TKDE.2018.2848260
Indexed BySCI
Language英语
Funding ProjectNatural Science Foundation of China[61333018] ; Natural Science Foundation of China[61673380]
Funding OrganizationNatural Science Foundation of China
WOS Research AreaComputer Science ; Engineering
WOS SubjectComputer Science, Artificial Intelligence ; Computer Science, Information Systems ; Engineering, Electrical & Electronic
WOS IDWOS:000466933000013
PublisherIEEE COMPUTER SOC
Citation statistics
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/24574
Collection中国科学院自动化研究所
Corresponding AuthorLi, Haoran
Affiliation1.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
2.Univ Chinese Acad Sci, Beijing 100190, Peoples R China
3.Chinese Acad Sci, Natl Lab Pattern Recognit, Inst Automat, Beijing 100864, Peoples R China
4.Chinese Acad Sci, CAS Ctr Excellence Brain Sci & Intelligence Techn, Beijing 100864, Peoples R China
5.Univ Chinese Acad Sci, Beijing 100049, Peoples R China
First Author AffilicationChinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
Corresponding Author AffilicationChinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
Recommended Citation
GB/T 7714
Li, Haoran,Zhu, Junnan,Ma, Cong,et al. Read, Watch, Listen, and Summarize: Multi-Modal Summarization for Asynchronous Text, Image, Audio and Video[J]. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,2019,31(5):996-1009.
APA Li, Haoran,Zhu, Junnan,Ma, Cong,Zhang, Jiajun,&Zong, Chengqing.(2019).Read, Watch, Listen, and Summarize: Multi-Modal Summarization for Asynchronous Text, Image, Audio and Video.IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING,31(5),996-1009.
MLA Li, Haoran,et al."Read, Watch, Listen, and Summarize: Multi-Modal Summarization for Asynchronous Text, Image, Audio and Video".IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING 31.5(2019):996-1009.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Li, Haoran]'s Articles
[Zhu, Junnan]'s Articles
[Ma, Cong]'s Articles
Baidu academic
Similar articles in Baidu academic
[Li, Haoran]'s Articles
[Zhu, Junnan]'s Articles
[Ma, Cong]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Li, Haoran]'s Articles
[Zhu, Junnan]'s Articles
[Ma, Cong]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.