Knowledge Commons of Institute of Automation，CAS
| Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective | |
Kong, Xiaoyu1,2; Deng, Yingying3 ; Tang, Fan4; Dong, Weiming3; Ma, Chongyang5; Chen, Yongyong; He, Zhenyu6,7; Xu, Changsheng3
| |
| Source Publication | IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
 |
| ISSN | 2162-237X |
| 2023-01-06 | |
| Pages | 15 |
| Corresponding Author | Tang, Fan(tfan.108@gmail.com) |
| Abstract | Arbitrary image stylization by neural networks has become a popular topic, and video stylization is attracting more attention as an extension of image stylization. However, when image stylization methods are applied to videos, unsatisfactory results that suffer from severe flickering effects appear. In this article, we conducted a detailed and comprehensive analysis of the cause of such flickering effects. Systematic comparisons among typical neural style transfer approaches show that the feature migration modules for state-of-the-art (SOTA) learning systems are ill-conditioned and could lead to a channelwise misalignment between the input content representations and the generated frames. Unlike traditional methods that relieve the misalignment via additional optical flow constraints or regularization modules, we focus on keeping the temporal consistency by aligning each output frame with the input frame. To this end, we propose a simple yet efficient multichannel correlation network (MCCNet), to ensure that output frames are directly aligned with inputs in the hidden feature space while maintaining the desired style patterns. An inner channel similarity loss is adopted to eliminate side effects caused by the absence of nonlinear operations such as softmax for strict alignment. Furthermore, to improve the performance of MCCNet under complex light conditions, we introduce an illumination loss during training. Qualitative and quantitative evaluations demonstrate that MCCNet performs well in arbitrary video and image style transfer tasks. |
| Keyword | Correlation Task analysis Optical imaging Integrated optics Lighting Optical fiber networks Image reconstruction Arbitrary stylization channel correlation cross-domain feature migration |
| DOI | 10.1109/TNNLS.2022.3230084 |
| Indexed By | SCI |
| Language | 英语 |
| Funding Project | National Key R&D Program of China[2020AAA0106200] ; National Science Foundation of China[U20B2070] ; National Science Foundation of China[61832016] ; National Science Foundation of China[62102162] ; National Science Foundation of China[62172126] ; National Science Foundation of China[6216063] ; Major Key Project of Peng Cheng Laboratory[PCL2021A03-1] ; Shenzhen Research Council[JCYJ20210324120202006] ; Guangdong Natural Science Foundation[2022A1515010819] |
| Funding Organization | National Key R&D Program of China ; National Science Foundation of China ; Major Key Project of Peng Cheng Laboratory ; Shenzhen Research Council ; Guangdong Natural Science Foundation |
| WOS Research Area | Computer Science ; Engineering |
| WOS Subject | Computer Science, Artificial Intelligence ; Computer Science, Hardware & Architecture ; Computer Science, Theory & Methods ; Engineering, Electrical & Electronic |
| WOS ID | WOS:000915636700001 |
| Publisher | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
| Citation statistics | |
| Document Type | 期刊论文 |
| Identifier | http://ir.ia.ac.cn/handle/173211/51423 |
| Collection | 多模态人工智能系统全国重点实验室 |
| Corresponding Author | Tang, Fan |
| Affiliation | 1.Jilin Univ, Sch Artificial Intelligence, Changchun 130012, Peoples R China 2.Harbin Inst Technol, Sch Comp Sci & Technol, Shenzhen 518073, Peoples R China 3.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China 4.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 5.Kuaishou Technol, Beijing 100085, Peoples R China 6.Harbin Inst Technol, Dept Comp Sci, Shenzhen 518073, Peoples R China 7.Peng Cheng Lab, Shenzhen 518055, Peoples R China |
| Recommended Citation GB/T 7714 | Kong, Xiaoyu,Deng, Yingying,Tang, Fan,et al. Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2023:15. |
| APA | Kong, Xiaoyu.,Deng, Yingying.,Tang, Fan.,Dong, Weiming.,Ma, Chongyang.,...&Xu, Changsheng.(2023).Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,15. |
| MLA | Kong, Xiaoyu,et al."Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS (2023):15. |
| Files in This Item: | There are no files associated with this item. | |||||
Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.
Edit Comment