CASIA OpenIR  > 智能感知与计算研究中心
Instance-aware Image and Sentence Matching with Selective Multimodal LSTM
Huang Yan(黄岩)1,2; Wang We(王威)1,2; Wang Liang(王亮)1,2
2017-08
Conference NameIEEE Conference on Computer Vision and Pattern Recognition
Conference Date2017.7.20
Conference PlaceUSA
Abstract
Effective image and sentence matching depends on how
to well measure their global visual-semantic similarity.
Based on the observation that such a global similarity arises
from a complex aggregation of multiple local similarities
between pairwise instances of image (objects) and sentence
(words), we propose a selective multimodal Long Short-
Term Memory network (sm-LSTM) for instance-aware image
and sentence matching. The sm-LSTM includes a multimodal
context-modulated attention scheme at each timestep
that can selectively attend to a pair of instances of image
and sentence, by predicting pairwise instance-aware
saliency maps for image and sentence. For selected pairwise
instances, their representations are obtained based on
the predicted saliency maps, and then compared to measure
their local similarity. By similarly measuring multiple local
similarities within a few timesteps, the sm-LSTM sequentially
aggregates them with hidden states to obtain a final
matching score as the desired global similarity. Extensive
experiments show that our model can well match image and
sentence with complex content, and achieve the state-of-theart
results on two public benchmark datasets.
Indexed ByEI
Document Type会议论文
Identifierhttp://ir.ia.ac.cn/handle/173211/14818
Collection智能感知与计算研究中心
Affiliation1.中国科学院自动化研究所
2.中国科学院大学
Recommended Citation
GB/T 7714
Huang Yan,Wang We,Wang Liang. Instance-aware Image and Sentence Matching with Selective Multimodal LSTM[C],2017.
Files in This Item: Download All
File Name/Size DocType Version Access License
Instance-aware Image(1692KB)会议论文 开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Huang Yan(黄岩)]'s Articles
[Wang We(王威)]'s Articles
[Wang Liang(王亮)]'s Articles
Baidu academic
Similar articles in Baidu academic
[Huang Yan(黄岩)]'s Articles
[Wang We(王威)]'s Articles
[Wang Liang(王亮)]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Huang Yan(黄岩)]'s Articles
[Wang We(王威)]'s Articles
[Wang Liang(王亮)]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: Instance-aware Image and Sentence Matching with Selective Multimodal LSTM.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.