CASIA OpenIR  > 模式识别国家重点实验室  > 先进数据分析与学习
Cross-Modal Retrieval via Deep and Bidirectional Representation Learning
He, Yonghao1; Xiang, Shiming1; Kang, Cuicui2; Wang, Jian1; Pan, Chunhong1; Xiang,Shiming
AbstractCross-modal retrieval emphasizes understanding inter-modality semantic correlations, which is often achieved by designing a similarity function. Generally, one of the most important things considered by the similarity function is how to make the cross-modal similarity computable. In this paper, a deep and bidirectional representation learning model is proposed to address the issue of image-text cross-modal retrieval. Owing to the solid progress of deep learning in computer vision and natural language processing, it is reliable to extract semantic representations from both raw image and text data by using deep neural networks. Therefore, in the proposed model, two convolution-based networks are adopted to accomplish representation learning for images and texts. By passing the networks, images and texts are mapped to a common space, in which the cross-modal similarity is measured by cosine distance. Subsequently, a bidirectional network architecture is designed to capture the property of the cross-modal retrieval-the bidirectional search. Such architecture is characterized by simultaneously involving the matched and unmatched image-text pairs for training. Accordingly, a learning framework with maximum likelihood criterion is finally developed. The network parameters are optimized via backpropagation and stochastic gradient descent. A great deal of experiments are conducted to sufficiently evaluate the proposed method on three publicly released datasets: IAPRTC-12, Flickr30k, and Flickr8k. The overall results definitely show that the proposed architecture is effective and the learned representations have good semantics to achieve superior cross-modal retrieval performance.
KeywordBidirectional Modeling Convolutional Neural Network Cross-modal Retrieval Representation Learning Word Embedding
WOS HeadingsScience & Technology ; Technology
Indexed BySCI
Funding OrganizationNational Basic Research Program of China(2012CB316304) ; Strategic Priority Research Program of the CAS(XDB02060009) ; National Natural Science Foundation of China(61272331 ; Beijing Natural Science Foundation(4162064) ; 91338202)
WOS Research AreaComputer Science ; Telecommunications
WOS SubjectComputer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS IDWOS:000379752600012
Citation statistics
Cited Times:16[WOS]   [WOS Record]     [Related Records in WOS]
Document Type期刊论文
Corresponding AuthorXiang,Shiming
Affiliation1.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
2.Chinese Acad Sci, Inst Informat Engn, Beijing 100093, Peoples R China
Recommended Citation
GB/T 7714
He, Yonghao,Xiang, Shiming,Kang, Cuicui,et al. Cross-Modal Retrieval via Deep and Bidirectional Representation Learning[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2016,18(7):1363-1377.
APA He, Yonghao,Xiang, Shiming,Kang, Cuicui,Wang, Jian,Pan, Chunhong,&Xiang,Shiming.(2016).Cross-Modal Retrieval via Deep and Bidirectional Representation Learning.IEEE TRANSACTIONS ON MULTIMEDIA,18(7),1363-1377.
MLA He, Yonghao,et al."Cross-Modal Retrieval via Deep and Bidirectional Representation Learning".IEEE TRANSACTIONS ON MULTIMEDIA 18.7(2016):1363-1377.
Files in This Item: Download All
File Name/Size DocType Version Access License
TMM2558463_final.pdf(11388KB)期刊论文作者接受稿开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[He, Yonghao]'s Articles
[Xiang, Shiming]'s Articles
[Kang, Cuicui]'s Articles
Baidu academic
Similar articles in Baidu academic
[He, Yonghao]'s Articles
[Xiang, Shiming]'s Articles
[Kang, Cuicui]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[He, Yonghao]'s Articles
[Xiang, Shiming]'s Articles
[Kang, Cuicui]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: TMM2558463_final.pdf
Format: Adobe PDF
All comments (0)
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.