CASIA OpenIR  > 模式识别国家重点实验室  > 自然语言处理
Multi-modal Sentence Summarization with Modality Attention and Image Filtering
Haoran Li1,2; Junnan Zhu1,2; Tianshang Liu1,2; Jiajun Zhang1,2; Chengqing Zong1,2,3
Conference NameProceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
Conference Date2018
Conference PlaceSweden

In this paper, we introduce a multi-modal sentence summarization task that produces a short summary from a pair of sentence and image. This task is more challenging than sentence summarization. It not only needs to effectively incorporate visual features into standard text summarization framework, but also requires to avoid noise of image. To this end, we propose a modality-based attention mechanism to pay different attention to image patches and text units, and we design image filters to selectively
use visual information to enhance the semantics of the input sentence. We construct a multimodal sentence summarization dataset and extensive
experiments on this dataset demonstrate that our models significantly outperform conventional models which only employ text as input. Further
analyses suggest that sentence summarization task can benefit from visually grounded representations from a variety of aspects.

Document Type会议论文
Affiliation1.National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences
2.University of Chinese Academy of Sciences
3.CAS Center for Excellence in Brain Science and Intelligence Technology
First Author AffilicationChinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
Recommended Citation
GB/T 7714
Haoran Li,Junnan Zhu,Tianshang Liu,et al. Multi-modal Sentence Summarization with Modality Attention and Image Filtering[C]:长文,2018.
Files in This Item: Download All
File Name/Size DocType Version Access License
3IJCAI.pdf(352KB)会议论文 开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Haoran Li]'s Articles
[Junnan Zhu]'s Articles
[Tianshang Liu]'s Articles
Baidu academic
Similar articles in Baidu academic
[Haoran Li]'s Articles
[Junnan Zhu]'s Articles
[Tianshang Liu]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Haoran Li]'s Articles
[Junnan Zhu]'s Articles
[Tianshang Liu]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: 3IJCAI.pdf
Format: Adobe PDF
All comments (0)
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.