CASIA OpenIR  > 模式识别实验室
DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulation
Yueming Lyu; Tianwei Lin; Fu Li; Dongliang He; Jing Dong; Tieniu Tan
2023
Conference NameIEEE Conference on Computer Vision and Pattern Recognition
Conference Date2023-06
Conference PlaceVancouver, Canada
Abstract

Text-driven image manipulation remains challenging in training or inference flexibility. Conditional generative models depend heavily on expensive annotated training data. Meanwhile, recent frameworks, which leverage pre-trained vision-language models, are limited by either per text-prompt optimization or inference-time hyperparameters tuning. In this work, we propose a novel framework named DeltaEdit to address these problems. Our key idea is to investigate and identify a space, namely delta image and text space that has well-aligned distribution between CLIP visual feature differences of two images and CLIP textual embedding differences of source and target texts. Based on the CLIP delta space, the DeltaEdit network is designed to map the CLIP visual features differences to the editing directions of StyleGAN at training phase. Then, in inference phase, DeltaEdit predicts the StyleGAN’s editing directions from the differences of the CLIP textual features. In this way, DeltaEdit is trained in a text-free manner. Once trained, it can well generalize to various text prompts for zero-shot inference without bells and whistles. Code is available at https://github.com/Yueming6568/DeltaEdit.

Sub direction classification图像视频处理与分析
planning direction of the national heavy laboratory视觉信息处理
Paper associated data
Document Type会议论文
Identifierhttp://ir.ia.ac.cn/handle/173211/56616
Collection模式识别实验室
Corresponding AuthorJing Dong
Affiliation1.Institute of Automation, Chinese Academy of Sciences
2.University of Chinese Academy of Sciences
3.Baidu Inc
4.Nanjing University
Recommended Citation
GB/T 7714
Yueming Lyu,Tianwei Lin,Fu Li,et al. DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulation[C],2023.
Files in This Item: Download All
File Name/Size DocType Version Access License
Lyu_DeltaEdit_Explor(4621KB)会议论文 开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Yueming Lyu]'s Articles
[Tianwei Lin]'s Articles
[Fu Li]'s Articles
Baidu academic
Similar articles in Baidu academic
[Yueming Lyu]'s Articles
[Tianwei Lin]'s Articles
[Fu Li]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Yueming Lyu]'s Articles
[Tianwei Lin]'s Articles
[Fu Li]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: Lyu_DeltaEdit_Exploring_Text-Free_Training_for_Text-Driven_Image_Manipulation_CVPR_2023_paper.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.