CASIA OpenIR  > 模式识别国家重点实验室  > 多媒体计算与图形学
Text2Video: An End-to-end Learning Framework for Expressing Text With Videos
Yang, Xiaoshan1,2; Zhang, Tianzhu1,2; Xu, Changsheng1,2
Source PublicationIEEE TRANSACTIONS ON MULTIMEDIA
2018-09-01
Volume20Issue:9Pages:2360-2370
SubtypeArticle
AbstractVideo creation is a challenging and highly professional task that generally involves substantial manual efforts. To ease this burden, a better approach is to automatically produce new videos based on clips from the massive amount of existing videos according to arbitrary text. In this paper, we formulate video creation as a problem of retrieving a sequence of videos for a sentence stream. To achieve this goal, we propose a novel multimodal recurrent architecture for automatic video production. Compared with existing methods, the proposed model has three major advantages. First, it is the first completely integrated end-to-end deep learning system for real-world production to the best of our knowledge. We are among the first to address the problem of retrieving a sequence of videos for a sentence stream. Second, it can effectively exploit the correspondence between sentences and video clips through semantic consistency modeling. Third, it can model the visual coherence well by requiring that the produced videos should be organized coherently in terms of visual appearance. We have conducted extensive experiments on two applications, including video retrieval and video composition. The qualitative and quantitative results obtained on two public datasets used in the Large Scale Movie Description Challenge 2016 both demonstrate the effectiveness of the proposed model compared with other state-of-the-art algorithms.
KeywordMultimedia Storytelling Video Analysis Deep Learning
WOS HeadingsScience & Technology ; Technology
DOI10.1109/TMM.2018.2807588
WOS KeywordANNOTATION ; REPRESENTATION ; NARRATIVES ; MOVIE ; WEB ; TV
Indexed BySCI
Language英语
Funding OrganizationNational Natural Science Foundation of China(61432019 ; Beijing Natural Science Foundation(4172062) ; Key Research Program of Frontier Sciences, CAS(QYZDJ-SSW-JSC039) ; 61572498 ; 61532009 ; 61702511 ; 61720106006 ; 61711530243)
WOS Research AreaComputer Science ; Telecommunications
WOS SubjectComputer Science, Information Systems ; Computer Science, Software Engineering ; Telecommunications
WOS IDWOS:000442358200010
Citation statistics
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/20467
Collection模式识别国家重点实验室_多媒体计算与图形学
Affiliation1.National Lab of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences
2.University of Chinese Academy of Sciences
Recommended Citation
GB/T 7714
Yang, Xiaoshan,Zhang, Tianzhu,Xu, Changsheng. Text2Video: An End-to-end Learning Framework for Expressing Text With Videos[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2018,20(9):2360-2370.
APA Yang, Xiaoshan,Zhang, Tianzhu,&Xu, Changsheng.(2018).Text2Video: An End-to-end Learning Framework for Expressing Text With Videos.IEEE TRANSACTIONS ON MULTIMEDIA,20(9),2360-2370.
MLA Yang, Xiaoshan,et al."Text2Video: An End-to-end Learning Framework for Expressing Text With Videos".IEEE TRANSACTIONS ON MULTIMEDIA 20.9(2018):2360-2370.
Files in This Item: Download All
File Name/Size DocType Version Access License
Text2Video.pdf(2281KB)期刊论文作者接受稿开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Yang, Xiaoshan]'s Articles
[Zhang, Tianzhu]'s Articles
[Xu, Changsheng]'s Articles
Baidu academic
Similar articles in Baidu academic
[Yang, Xiaoshan]'s Articles
[Zhang, Tianzhu]'s Articles
[Xu, Changsheng]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Yang, Xiaoshan]'s Articles
[Zhang, Tianzhu]'s Articles
[Xu, Changsheng]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: Text2Video.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.