Knowledge Commons of Institute of Automation,CAS
Show, Tell, and Polish: Ruminant Decoding for Image Captioning | |
Guo, Longteng1,2![]() ![]() ![]() | |
发表期刊 | IEEE TRANSACTIONS ON MULTIMEDIA
![]() |
ISSN | 1520-9210 |
2020-08-01 | |
卷号 | 22期号:8页码:2149-2162 |
摘要 | The encoder-decoder framework has been the base of popular image captioning models, which typically predicts the target sentence based on the encoded source image one word at a time in sequence. However, such a single-pass decoding framework encounters two problems. First, mistakes in the predicted words cannot be corrected and may propagate to the entire sentence. Second, because the single-pass decoder cannot access the following un-generated words, it can only perform local planning to choose every single word according to the preceding words, while lacks the global planning ability as for maintaining the semantic consistency and fluency of the whole sentence. In order to address the above two problems, in this work, we design a ruminant captioning framework which contains an image encoder, a base decoder, and a ruminant decoder. Specifically, the outputs of the former/base decoder are utilized as the global information to guide the words prediction of the latter/ruminant decoder, in an attempt to mimic human polishing process. We enable jointly training of the whole framework and overcome the non-differential problem of discrete words by designing a novel reinforcement learning based optimization algorithm. Experiments on two datasets (MS COCO and Flickr30 k) demonstrate that our ruminant decoding method can bring significant improvements over traditional single-pass decoding based models and achieves state-of-the-art performance. |
关键词 | Image captioning Multi-pass decoding Rumination |
DOI | 10.1109/TMM.2019.2951226 |
收录类别 | SCI |
语种 | 英语 |
资助项目 | National Natural Science Foundation of China[61922086] ; National Natural Science Foundation of China[61872366] ; Beijing Natural Science Foundation[4192059] |
项目资助者 | National Natural Science Foundation of China ; Beijing Natural Science Foundation |
WOS研究方向 | Computer Science ; Telecommunications |
WOS类目 | Computer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications |
WOS记录号 | WOS:000553424500019 |
出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
七大方向——子方向分类 | 多模态智能 |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/40260 |
专题 | 紫东太初大模型研究中心_图像与视频分析 |
通讯作者 | Liu, Jing |
作者单位 | 1.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Wuhan Univ, Sch Comp, Wuhan 430072, Peoples R China |
第一作者单位 | 模式识别国家重点实验室 |
通讯作者单位 | 模式识别国家重点实验室 |
推荐引用方式 GB/T 7714 | Guo, Longteng,Liu, Jing,Lu, Shichen,et al. Show, Tell, and Polish: Ruminant Decoding for Image Captioning[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2020,22(8):2149-2162. |
APA | Guo, Longteng,Liu, Jing,Lu, Shichen,&Lu, Hanqing.(2020).Show, Tell, and Polish: Ruminant Decoding for Image Captioning.IEEE TRANSACTIONS ON MULTIMEDIA,22(8),2149-2162. |
MLA | Guo, Longteng,et al."Show, Tell, and Polish: Ruminant Decoding for Image Captioning".IEEE TRANSACTIONS ON MULTIMEDIA 22.8(2020):2149-2162. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
3. [TMM.2019] Show, (4378KB) | 期刊论文 | 作者接受稿 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论