CASIA OpenIR  > 多模态人工智能系统全国重点实验室
ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models
Zhang, Yuxin1,2; Dong, Weiming1,2; Tang, Fan3; Huang, Nisha1,2; Huang, Haibin4; Ma, Chongyang4; Lee, Tong-Yee5; Deussen, Oliver6; Xu, Changsheng1,2
Source PublicationACM TRANSACTIONS ON GRAPHICS
ISSN0730-0301
2023-12-01
Volume42Issue:6Pages:14
Corresponding AuthorDong, Weiming(weiming.dong@ia.ac.cn)
AbstractPersonalizing generative models offers a way to guide image generation with user-provided references. Current personalization methods can invert an object or concept into the textual conditioning space and compose new natural sentences for text-to-image diffusion models. However, representing and editing specific visual attributes such as material, style, and layout remains a challenge, leading to a lack of disentanglement and editability. To address this problem, we propose a novel approach that leverages the step-by-step generation process of diffusion models, which generate images from low to high frequency information, providing a new perspective on representing, generating, and editing images. We develop the Prompt Spectrum Space P*, an expanded textual conditioning space, and a new image representation method called ProSpect. ProSpect represents an image as a collection of inverted textual token embeddings encoded from per-stage prompts, where each prompt corresponds to a specific generation stage (i.e., a group of consecutive steps) of the diffusion model. Experimental results demonstrate that P* and ProSpect offer better disentanglement and controllability compared to existing methods. We apply ProSpect in various personalized attribute-aware image generation applications, such as image-guided or text-driven manipulations of materials, style, and layout, achieving previously unattainable results from a single image input without fine-tuning the diffusion models. Our source code is available at https: //github.com/zyxElsa/ProSpect.
KeywordImage generation Diffusion models Attribute-aware editing Model personalization
DOI10.1145/3618342
Indexed BySCI
Language英语
Funding ProjectNational Key R&D Program of China[2020AAA0106200] ; National Natural Science Foundation of China[61832016] ; National Natural Science Foundation of China[62102162] ; National Natural Science Foundation of China[U20B2070] ; Beijing Natural Science Foundation[L221013] ; National Science and Technology Council[111-2221-E-006-112-MY3] ; Deutsche Forschungsgemeinschaft (DFG)[413891298]
Funding OrganizationNational Key R&D Program of China ; National Natural Science Foundation of China ; Beijing Natural Science Foundation ; National Science and Technology Council ; Deutsche Forschungsgemeinschaft (DFG)
WOS Research AreaComputer Science
WOS SubjectComputer Science, Software Engineering
WOS IDWOS:001139790400072
PublisherASSOC COMPUTING MACHINERY
Citation statistics
Cited Times:1[WOS]   [WOS Record]     [Related Records in WOS]
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/55395
Collection多模态人工智能系统全国重点实验室
Corresponding AuthorDong, Weiming
Affiliation1.Chinese Acad Sci, Inst Automat, MAIS, Beijing, Peoples R China
2.UCAS, Sch Artificial Intelligence, Beijing, Peoples R China
3.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China
4.Kuaishou Technol, Beijing, Peoples R China
5.Natl Cheng Kung Univ, Tainan, Taiwan
6.Univ Konstanz, Constance, Germany
First Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Corresponding Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
Zhang, Yuxin,Dong, Weiming,Tang, Fan,et al. ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models[J]. ACM TRANSACTIONS ON GRAPHICS,2023,42(6):14.
APA Zhang, Yuxin.,Dong, Weiming.,Tang, Fan.,Huang, Nisha.,Huang, Haibin.,...&Xu, Changsheng.(2023).ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models.ACM TRANSACTIONS ON GRAPHICS,42(6),14.
MLA Zhang, Yuxin,et al."ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models".ACM TRANSACTIONS ON GRAPHICS 42.6(2023):14.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Zhang, Yuxin]'s Articles
[Dong, Weiming]'s Articles
[Tang, Fan]'s Articles
Baidu academic
Similar articles in Baidu academic
[Zhang, Yuxin]'s Articles
[Dong, Weiming]'s Articles
[Tang, Fan]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Zhang, Yuxin]'s Articles
[Dong, Weiming]'s Articles
[Tang, Fan]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.