M3: Multimodal Memory Modelling for Video Captioning | |
Wang, Junbo![]() ![]() ![]() ![]() | |
2018 | |
会议名称 | IEEE Conference on Computer Vision and Pattern Recognition |
会议日期 | 2018-6 |
会议地点 | Salt Lake City, USA |
摘要 | Video captioning which automatically translates video clips into natural language sentences is a very important task in computer vision. By virtue of recent deep learning technologies, video captioning has made great progress. However, learning an effective mapping from the visual sequence space to the language space is still a challenging problem due to the long-term multimodal dependency modelling and semantic misalignment. Inspired by the facts that memory modelling poses potential advantages to long-term sequential problems [35] and working memory is the key factor of visual attention [33], we propose a Multi-modal Memory Model (M3) to describe videos, which build- s a visual and textual shared memory to model the long-term visual-textual dependency and further guide visual attention on described visual targets to solve visual-textual alignments. Specifically, similar to [10], the proposed M3 attaches an external memory to store and retrieve both visual and textual contents by interacting with video and sentence with multiple read and write operations. To evaluate the proposed model, we perform experiments on two public datasets: MSVD and MSR-VTT. The experimental results demonstrate that our method outperforms most of the state- of-the-art methods in terms of BLEU and METEOR. |
文献类型 | 会议论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/28358 |
专题 | 模式识别实验室 |
作者单位 | 1.Center for Research on Intelligent Perception and Computing, National Laboratory of Pattern Recognition 2.Center for Excellence in Brain Science and Intelligence Technology, Institute of Automation, Chinese Academy of Sciences 3.University of Chinese Academy of Sciences |
推荐引用方式 GB/T 7714 | Wang, Junbo,Wang, Wei,Huang, Yan,et al. M3: Multimodal Memory Modelling for Video Captioning[C],2018. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
CVPR2018.pdf(661KB) | 会议论文 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论