Instance-aware Image and Sentence Matching with Selective Multimodal LSTM

CASIA OpenIR > 智能感知与计算研究中心

	Instance-aware Image and Sentence Matching with Selective Multimodal LSTM
	Huang Yan(黄岩)1,2 ; Wang We(王威)1,2; Wang Liang(王亮)1,2
	2017-08
会议名称	IEEE Conference on Computer Vision and Pattern Recognition
会议日期	2017.7.20
会议地点	USA
摘要	Effective image and sentence matching depends on how to well measure their global visual-semantic similarity. Based on the observation that such a global similarity arises from a complex aggregation of multiple local similarities between pairwise instances of image (objects) and sentence (words), we propose a selective multimodal Long Short- Term Memory network (sm-LSTM) for instance-aware image and sentence matching. The sm-LSTM includes a multimodal context-modulated attention scheme at each timestep that can selectively attend to a pair of instances of image and sentence, by predicting pairwise instance-aware saliency maps for image and sentence. For selected pairwise instances, their representations are obtained based on the predicted saliency maps, and then compared to measure their local similarity. By similarly measuring multiple local similarities within a few timesteps, the sm-LSTM sequentially aggregates them with hidden states to obtain a final matching score as the desired global similarity. Extensive experiments show that our model can well match image and sentence with complex content, and achieve the state-of-theart results on two public benchmark datasets.
收录类别	EI
文献类型	会议论文
条目标识符	http://ir.ia.ac.cn/handle/173211/14818
专题	智能感知与计算研究中心
作者单位	1.中国科学院自动化研究所 2.中国科学院大学
第一作者单位	中国科学院自动化研究所
推荐引用方式 GB/T 7714	Huang Yan,Wang We,Wang Liang. Instance-aware Image and Sentence Matching with Selective Multimodal LSTM[C],2017.

条目包含的文件		下载所有文件
文件名称/大小	文献类型	版本类型	开放类型	使用许可
Instance-aware Image（1692KB）	会议论文		开放获取	CC BY-NC-SA	浏览下载