Image and Sentence Matching via Semantic Concepts and Order Learning | |
Huang, Yan![]() ![]() ![]() | |
发表期刊 | IEEE Transactions on Pattern Analysis and Machine Intelligence(IEEE TPAMI)
![]() |
ISSN | 0162-8828 |
2019 | |
卷号 | 0期号:0页码:0-0 |
通讯作者 | Huang, Yan(yhuang@nlpr.ia.ac.cn) |
文章类型 | regular paper |
摘要 | Image and sentence matching remains challenging due to the large visual-semantic discrepancy. This mainly arises from two aspects: 1) images consist of unstructured content which is not semantically abstract as the words in the sentences, so they are not directly comparable, and 2) arranging semantic concepts in different semantic order could lead to quite diverse meanings. In this work, we propose a semantic concepts and order learning framework, which can improve the image representation by first predicting semantic concepts and then organizing them in a correct semantic order. Given an image, we first use a multi-regional multi-label CNN to predict its included semantic concepts in terms of object, property and action. Then, to organize these concepts, we use a context-modulated attentional LSTM to learn the semantic order. It regards the predicted semantic concepts and image global scene as context at each timestep, and selectively attends to concept-related image regions in a sequential order. To further enhance the semantic order, we perform an additional sentence generation. We learn the sentence representation with a conventional LSTM, and then jointly perform image and sentence matching and sentence generation for model learning. Extensive experiments demonstrate that our model can achieve the state-of-the-art results on two public benchmark datasets. |
关键词 | Image And Sentence Matching |
学科门类 | 工学 |
DOI | 10.1109/TPAMI.2018.2883466 |
URL | 查看原文 |
收录类别 | SCI |
语种 | 英语 |
资助项目 | National Key Research and Development Program of China[2016YFB1001000] ; National Natural Science Foundation of China[61525306] ; National Natural Science Foundation of China[61633021] ; National Natural Science Foundation of China[61721004] ; National Natural Science Foundation of China[61420106015] ; Beijing Natural Science Foundation[4162058] ; Capital Science and Technology Leading Talent Training Project[Z181100006318030] ; Beijing Science and Technology Project[Z181100008918010] ; Chinese Academy Sciences Artificial Intelligence Research (CAS-AIR) |
项目资助者 | National Key Research and Development Program of China ; National Natural Science Foundation of China ; Beijing Natural Science Foundation ; Capital Science and Technology Leading Talent Training Project ; Beijing Science and Technology Project ; Chinese Academy Sciences Artificial Intelligence Research (CAS-AIR) |
WOS研究方向 | Computer Science ; Engineering |
WOS类目 | Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic |
WOS记录号 | WOS:000525365300009 |
出版者 | IEEE COMPUTER SOC |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/25796 |
专题 | 模式识别实验室 |
通讯作者 | Huang, Yan |
推荐引用方式 GB/T 7714 | Huang, Yan,Wu, Qi,Wang, Wei,et al. Image and Sentence Matching via Semantic Concepts and Order Learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence(IEEE TPAMI),2019,0(0):0-0. |
APA | Huang, Yan,Wu, Qi,Wang, Wei,&Wang, Liang.(2019).Image and Sentence Matching via Semantic Concepts and Order Learning.IEEE Transactions on Pattern Analysis and Machine Intelligence(IEEE TPAMI),0(0),0-0. |
MLA | Huang, Yan,et al."Image and Sentence Matching via Semantic Concepts and Order Learning".IEEE Transactions on Pattern Analysis and Machine Intelligence(IEEE TPAMI) 0.0(2019):0-0. |
条目包含的文件 | 条目无相关文件。 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论