CASIA OpenIR
Image Captioning with Word Gate and Adaptive Self-Critical Learning
Zhu, Xinxin1,2; Li, Lixiang1,2; Liu, Jing3; Guo, Longteng3; Fang, Zhiwei3; Peng, Haipeng1,2; Niu, Xinxin1,2
Source PublicationAPPLIED SCIENCES-BASEL
ISSN2076-3417
2018-06-01
Volume8Issue:6Pages:13
Corresponding AuthorLi, Lixiang(li_lixiang2006@163.com)
AbstractAlthough the policy-gradient methods for reinforcement learning have shown significant improvement in image captioning, how to achieve high performance during the reinforcement optimizing process is still not a simple task. There are at least two difficulties: (1) The large size of vocabulary leads to a large action space, which makes it difficult for the model to accurately predict the current word. (2) The large variance of gradient estimation in reinforcement learning usually causes severe instabilities in the training process. In this paper, we propose two innovations to boost the performance of self-critical sequence training (SCST). First, we modify the standard long short-term memory (LSTM)based decoder by introducing a gate function to reduce the search scope of the vocabulary for any given image, which is termed the word gate decoder. Second, instead of only considering current maximum actions greedily, we propose a stabilized gradient estimation method whose gradient variance is controlled by the difference between the sampling reward from the current model and the expectation of the historical reward. We conducted extensive experiments, and results showed that our method could accelerate the training process and increase the prediction accuracy. Our method was validated on MS COCO datasets and yielded state-of-the-art performance.
Keywordimage caption image understanding deep learning computer vision
DOI10.3390/app8060909
Indexed BySCI
Language英语
Funding ProjectNational Key Research and Development Program of China[2016YFB0800602] ; National Natural Science Foundation of China[61771071] ; National Natural Science Foundation of China[61573067]
Funding OrganizationNational Key Research and Development Program of China ; National Natural Science Foundation of China
WOS Research AreaChemistry ; Materials Science ; Physics
WOS SubjectChemistry, Multidisciplinary ; Materials Science, Multidisciplinary ; Physics, Applied
WOS IDWOS:000436488000068
PublisherMDPI
Citation statistics
Cited Times:2[WOS]   [WOS Record]     [Related Records in WOS]
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/26325
Collection中国科学院自动化研究所
Corresponding AuthorLi, Lixiang
Affiliation1.Beijing Univ Posts & Telecommun, Informat Secur Ctr, State Key Lab Networking & Switching Technol, Beijing 100876, Peoples R China
2.Beijing Univ Posts & Telecommun, Natl Engn Lab Disaster Backup & Recovery, Beijing 100876, Peoples R China
3.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
Recommended Citation
GB/T 7714
Zhu, Xinxin,Li, Lixiang,Liu, Jing,et al. Image Captioning with Word Gate and Adaptive Self-Critical Learning[J]. APPLIED SCIENCES-BASEL,2018,8(6):13.
APA Zhu, Xinxin.,Li, Lixiang.,Liu, Jing.,Guo, Longteng.,Fang, Zhiwei.,...&Niu, Xinxin.(2018).Image Captioning with Word Gate and Adaptive Self-Critical Learning.APPLIED SCIENCES-BASEL,8(6),13.
MLA Zhu, Xinxin,et al."Image Captioning with Word Gate and Adaptive Self-Critical Learning".APPLIED SCIENCES-BASEL 8.6(2018):13.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Zhu, Xinxin]'s Articles
[Li, Lixiang]'s Articles
[Liu, Jing]'s Articles
Baidu academic
Similar articles in Baidu academic
[Zhu, Xinxin]'s Articles
[Li, Lixiang]'s Articles
[Liu, Jing]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Zhu, Xinxin]'s Articles
[Li, Lixiang]'s Articles
[Liu, Jing]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.