Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective

doi:10.1109/TNNLS.2022.3230084

	Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective
	Kong, Xiaoyu 1,2; Deng, Yingying3 ; Tang, Fan4 ; Dong, Weiming3 ; Ma, Chongyang 5; Chen, Yongyong ; He, Zhenyu 6,7; Xu, Changsheng3
发表期刊	IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
ISSN	2162-237X
	2023-01-06
页码	15
通讯作者	Tang, Fan(tfan.108@gmail.com)
摘要	Arbitrary image stylization by neural networks has become a popular topic, and video stylization is attracting more attention as an extension of image stylization. However, when image stylization methods are applied to videos, unsatisfactory results that suffer from severe flickering effects appear. In this article, we conducted a detailed and comprehensive analysis of the cause of such flickering effects. Systematic comparisons among typical neural style transfer approaches show that the feature migration modules for state-of-the-art (SOTA) learning systems are ill-conditioned and could lead to a channelwise misalignment between the input content representations and the generated frames. Unlike traditional methods that relieve the misalignment via additional optical flow constraints or regularization modules, we focus on keeping the temporal consistency by aligning each output frame with the input frame. To this end, we propose a simple yet efficient multichannel correlation network (MCCNet), to ensure that output frames are directly aligned with inputs in the hidden feature space while maintaining the desired style patterns. An inner channel similarity loss is adopted to eliminate side effects caused by the absence of nonlinear operations such as softmax for strict alignment. Furthermore, to improve the performance of MCCNet under complex light conditions, we introduce an illumination loss during training. Qualitative and quantitative evaluations demonstrate that MCCNet performs well in arbitrary video and image style transfer tasks.
关键词	Correlation Task analysis Optical imaging Integrated optics Lighting Optical fiber networks Image reconstruction Arbitrary stylization channel correlation cross-domain feature migration
DOI	10.1109/TNNLS.2022.3230084
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China[2020AAA0106200] ; National Science Foundation of China[U20B2070] ; National Science Foundation of China[61832016] ; National Science Foundation of China[62102162] ; National Science Foundation of China[62172126] ; National Science Foundation of China[6216063] ; Major Key Project of Peng Cheng Laboratory[PCL2021A03-1] ; Shenzhen Research Council[JCYJ20210324120202006] ; Guangdong Natural Science Foundation[2022A1515010819]
项目资助者	National Key R&D Program of China ; National Science Foundation of China ; Major Key Project of Peng Cheng Laboratory ; Shenzhen Research Council ; Guangdong Natural Science Foundation
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Hardware & Architecture ; Computer Science, Theory & Methods ; Engineering, Electrical & Electronic
WOS记录号	WOS:000915636700001
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：7[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/51423
专题	多模态人工智能系统全国重点实验室
通讯作者	Tang, Fan
作者单位	1.Jilin Univ, Sch Artificial Intelligence, Changchun 130012, Peoples R China 2.Harbin Inst Technol, Sch Comp Sci & Technol, Shenzhen 518073, Peoples R China 3.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China 4.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 5.Kuaishou Technol, Beijing 100085, Peoples R China 6.Harbin Inst Technol, Dept Comp Sci, Shenzhen 518073, Peoples R China 7.Peng Cheng Lab, Shenzhen 518055, Peoples R China
推荐引用方式 GB/T 7714	Kong, Xiaoyu,Deng, Yingying,Tang, Fan,et al. Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2023:15.
APA	Kong, Xiaoyu.,Deng, Yingying.,Tang, Fan.,Dong, Weiming.,Ma, Chongyang.,...&Xu, Changsheng.(2023).Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,15.
MLA	Kong, Xiaoyu,et al."Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS (2023):15.