CASIA OpenIR  > 智能感知与计算研究中心
Instance-aware Image and Sentence Matching with Selective Multimodal LSTM
Huang Yan(黄岩)1,2; Wang We(王威)1,2; Wang Liang(王亮)1,2
2017-08
会议名称IEEE Conference on Computer Vision and Pattern Recognition
会议日期2017.7.20
会议地点USA
摘要
Effective image and sentence matching depends on how
to well measure their global visual-semantic similarity.
Based on the observation that such a global similarity arises
from a complex aggregation of multiple local similarities
between pairwise instances of image (objects) and sentence
(words), we propose a selective multimodal Long Short-
Term Memory network (sm-LSTM) for instance-aware image
and sentence matching. The sm-LSTM includes a multimodal
context-modulated attention scheme at each timestep
that can selectively attend to a pair of instances of image
and sentence, by predicting pairwise instance-aware
saliency maps for image and sentence. For selected pairwise
instances, their representations are obtained based on
the predicted saliency maps, and then compared to measure
their local similarity. By similarly measuring multiple local
similarities within a few timesteps, the sm-LSTM sequentially
aggregates them with hidden states to obtain a final
matching score as the desired global similarity. Extensive
experiments show that our model can well match image and
sentence with complex content, and achieve the state-of-theart
results on two public benchmark datasets.
收录类别EI
文献类型会议论文
条目标识符http://ir.ia.ac.cn/handle/173211/14818
专题智能感知与计算研究中心
作者单位1.中国科学院自动化研究所
2.中国科学院大学
推荐引用方式
GB/T 7714
Huang Yan,Wang We,Wang Liang. Instance-aware Image and Sentence Matching with Selective Multimodal LSTM[C],2017.
条目包含的文件 下载所有文件
文件名称/大小 文献类型 版本类型 开放类型 使用许可
Instance-aware Image(1692KB)会议论文 开放获取CC BY-NC-SA浏览 下载
个性服务
推荐该条目
保存到收藏夹
查看访问统计
导出为Endnote文件
谷歌学术
谷歌学术中相似的文章
[Huang Yan(黄岩)]的文章
[Wang We(王威)]的文章
[Wang Liang(王亮)]的文章
百度学术
百度学术中相似的文章
[Huang Yan(黄岩)]的文章
[Wang We(王威)]的文章
[Wang Liang(王亮)]的文章
必应学术
必应学术中相似的文章
[Huang Yan(黄岩)]的文章
[Wang We(王威)]的文章
[Wang Liang(王亮)]的文章
相关权益政策
暂无数据
收藏/分享
文件名: Instance-aware Image and Sentence Matching with Selective Multimodal LSTM.pdf
格式: Adobe PDF
所有评论 (0)
暂无评论
 

除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。