Recall What You See Continually Using GridLSTM in Image Captioning

doi:10.1109/TMM.2019.2931815

	Recall What You See Continually Using GridLSTM in Image Captioning
	Wu, Lingxiang 1; Xu, Min 1; Wang, Jinqiao2 ; Perry, Stuart 3
发表期刊	IEEE TRANSACTIONS ON MULTIMEDIA
ISSN	1520-9210
	2020-03-01
卷号	22 期号:3 页码:808-818
通讯作者	Xu, Min(Min.Xu@uts.edu.au)
摘要	The goal of image captioning is to automatically describe an image with a sentence, and the task has attracted research attention from both the computer vision and natural-language processing research communities. The existing encoder-decoder model and its variants, which are the most popular models for image captioning, use the image features in three ways: first, they inject the encoded image features into the decoder only once at the initial step, which does not enable the rich image content to be explored sufficiently while gradually generating a text caption; second, they concatenate the encoded image features with text as extra inputs at every step, which introduces unnecessary noise; and, third, they using an attention mechanism, which increases the computational complexity due to the introduction of extra neural nets to identify the attention regions. Different from the existing methods, in this paper, we propose a novel network, Recall Network, for generating captions that are consistent with the images. The recall network selectively involves the visual features by using a GridLSTM and, thus, is able to recall image contents while generating each word. By importing the visual information as the latent memory along the depth dimension LSTM, the decoder is able to admit the visual features dynamically through the inherent LSTM structure without adding any extra neural nets or parameters. The Recall Network efficiently prevents the decoder from deviating from the original image content. To verify the efficiency of our model, we conducted exhaustive experiments on full and dense image captioning. The experimental results clearly demonstrate that our recall network outperforms the conventional encoder-decoder model by a large margin and that it performs comparably to the state-of-the-art methods.
关键词	Visualization Decoding Task analysis Neural networks Training Computational modeling Logic gates Image captioning GridLSTM recurrent neural network
DOI	10.1109/TMM.2019.2931815
关键词[WOS]	CLASSIFICATION ; ATTENTION
收录类别	SCI
语种	英语
WOS研究方向	Computer Science ; Telecommunications
WOS类目	Computer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS记录号	WOS:000519576700019
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：28[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/38640
专题	紫东太初大模型研究中心_图像与视频分析
通讯作者	Xu, Min
作者单位	1.Univ Technol Sydney, Global Big Data Technol Ctr, Sch Elect & Data Engn, Ultimo, NSW 2007, Australia 2.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China 3.Univ Technol Sydney, Sch Elect & Data Engn, Perceptual Imaging Lab, Ultimo, NSW 2007, Australia
推荐引用方式 GB/T 7714	Wu, Lingxiang,Xu, Min,Wang, Jinqiao,et al. Recall What You See Continually Using GridLSTM in Image Captioning[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2020,22(3):808-818.
APA	Wu, Lingxiang,Xu, Min,Wang, Jinqiao,&Perry, Stuart.(2020).Recall What You See Continually Using GridLSTM in Image Captioning.IEEE TRANSACTIONS ON MULTIMEDIA,22(3),808-818.
MLA	Wu, Lingxiang,et al."Recall What You See Continually Using GridLSTM in Image Captioning".IEEE TRANSACTIONS ON MULTIMEDIA 22.3(2020):808-818.