Learning Semantic Concepts and Order for Image and Sentence Matching | |
Huang, Yan; Wu, Qi; Wang, Liang | |
2018-06 | |
会议名称 | IEEE Conference on Computer Vision and Pattern Recognition (CVPR) |
卷号 | 0 |
期号 | 0 |
页码 | 6163-6171 |
会议日期 | 2018.6.18-2018.6.22 |
会议地点 | Salt Lake City |
会议录编者/会议主办者 | Michael Brown |
出版地 | USA |
出版者 | IEEE |
摘要 | Image and sentence matching has made great progress recently, but it remains challenging due to the large visual semantic discrepancy. This mainly arises from that the representation of pixel-level image usually lacks of high-level semantic information as in its matched sentence. In this work, we propose a semantic-enhanced image and sentence matching model, which can improve the image representation by learning semantic concepts and then organizing them in a correct semantic order. Given an image, we first use a multi-regional multi-label CNN to predict its semantic concepts, including objects, properties, actions, etc. Then, considering that different orders of semantic concepts lead to diverse semantic meanings, we use a context-gated sentence generation scheme for semantic order learning. It simultaneously uses the image global context containing concept relations as reference and the groundtruth semantic order in the matched sentence as supervision. After obtaining the improved image representation, we learn the sentence representation with a conventional LSTM, and then jointly perform image and sentence matching and sentence generation for model learning. Extensive experiments demonstrate the effectiveness of our learned semantic concepts and order, by achieving the state-of-the-art results on two public benchmark datasets. |
关键词 | Image And Sentence Matching |
学科门类 | 工学 |
DOI | 0 |
URL | 查看原文 |
收录类别 | EI |
语种 | 英语 |
引用统计 | |
文献类型 | 会议论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/25799 |
专题 | 模式识别实验室 |
作者单位 | 中科院自动化所 |
推荐引用方式 GB/T 7714 | Huang, Yan,Wu, Qi,Wang, Liang. Learning Semantic Concepts and Order for Image and Sentence Matching[C]//Michael Brown. USA:IEEE,2018:6163-6171. |
条目包含的文件 | 条目无相关文件。 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论