Recall What You See Continually Using GridLSTM in Image Captioning
Wu, Lingxiang1; Xu, Min1; Wang, Jinqiao2; Perry, Stuart3
Corresponding AuthorXu, Min(
AbstractThe goal of image captioning is to automatically describe an image with a sentence, and the task has attracted research attention from both the computer vision and natural-language processing research communities. The existing encoder-decoder model and its variants, which are the most popular models for image captioning, use the image features in three ways: first, they inject the encoded image features into the decoder only once at the initial step, which does not enable the rich image content to be explored sufficiently while gradually generating a text caption; second, they concatenate the encoded image features with text as extra inputs at every step, which introduces unnecessary noise; and, third, they using an attention mechanism, which increases the computational complexity due to the introduction of extra neural nets to identify the attention regions. Different from the existing methods, in this paper, we propose a novel network, Recall Network, for generating captions that are consistent with the images. The recall network selectively involves the visual features by using a GridLSTM and, thus, is able to recall image contents while generating each word. By importing the visual information as the latent memory along the depth dimension LSTM, the decoder is able to admit the visual features dynamically through the inherent LSTM structure without adding any extra neural nets or parameters. The Recall Network efficiently prevents the decoder from deviating from the original image content. To verify the efficiency of our model, we conducted exhaustive experiments on full and dense image captioning. The experimental results clearly demonstrate that our recall network outperforms the conventional encoder-decoder model by a large margin and that it performs comparably to the state-of-the-art methods.
KeywordVisualization Decoding Task analysis Neural networks Training Computational modeling Logic gates Image captioning GridLSTM recurrent neural network
Indexed BySCI
WOS Research AreaComputer Science ; Telecommunications
WOS SubjectComputer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS IDWOS:000519576700019
Citation statistics
Document Type期刊论文
Corresponding AuthorXu, Min
Affiliation1.Univ Technol Sydney, Global Big Data Technol Ctr, Sch Elect & Data Engn, Ultimo, NSW 2007, Australia
2.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
3.Univ Technol Sydney, Sch Elect & Data Engn, Perceptual Imaging Lab, Ultimo, NSW 2007, Australia
Recommended Citation
GB/T 7714
Wu, Lingxiang,Xu, Min,Wang, Jinqiao,et al. Recall What You See Continually Using GridLSTM in Image Captioning[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2020,22(3):808-818.
APA Wu, Lingxiang,Xu, Min,Wang, Jinqiao,&Perry, Stuart.(2020).Recall What You See Continually Using GridLSTM in Image Captioning.IEEE TRANSACTIONS ON MULTIMEDIA,22(3),808-818.
MLA Wu, Lingxiang,et al."Recall What You See Continually Using GridLSTM in Image Captioning".IEEE TRANSACTIONS ON MULTIMEDIA 22.3(2020):808-818.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Wu, Lingxiang]'s Articles
[Xu, Min]'s Articles
[Wang, Jinqiao]'s Articles
Baidu academic
Similar articles in Baidu academic
[Wu, Lingxiang]'s Articles
[Xu, Min]'s Articles
[Wang, Jinqiao]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Wu, Lingxiang]'s Articles
[Xu, Min]'s Articles
[Wang, Jinqiao]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.