CASIA OpenIR  > 模式识别国家重点实验室  > 自然语言处理
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Li, Haoran; Zhu, Junnan; Liu, Tianshang; Zhang, Jiajun; Zong, Chengqing
2018
Conference NameIJCAI
Conference Date2018-7
Conference PlaceStockholm, Sweden
Abstract

In this paper, we introduce a multi-modal sentence summarization task that produces a short summary from a pair of sentence and image. This task is more challenging than sentence summarization. It not only needs to effectively incorporate visual features into standard text summarization framework, but also requires to avoid noise of image. To this end, we propose a modality-based attention mechanism to pay different attention to image patches and text units, and we design image filters to selectively use visual information to enhance the semantics of the input sentence. We construct a multimodal sentence summarization dataset and extensive experiments on this dataset demonstrate that our models significantly outperform conventional models which only employ text as input. Further analyses suggest that sentence summarization task can benefit from visually grounded representations from a variety of aspects.

Document Type会议论文
Identifierhttp://ir.ia.ac.cn/handle/173211/23203
Collection模式识别国家重点实验室_自然语言处理
Affiliation中国科学院自动化研究所
First Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
Li, Haoran,Zhu, Junnan,Liu, Tianshang,et al. Multi-modal Sentence Summarization with Modality Attention and Image Filtering[C],2018.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Li, Haoran]'s Articles
[Zhu, Junnan]'s Articles
[Liu, Tianshang]'s Articles
Baidu academic
Similar articles in Baidu academic
[Li, Haoran]'s Articles
[Zhu, Junnan]'s Articles
[Liu, Tianshang]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Li, Haoran]'s Articles
[Zhu, Junnan]'s Articles
[Liu, Tianshang]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.