ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models

doi:10.1145/3618342

	ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models
	Zhang, Yuxin 1,2; Dong, Weiming1,2 ; Tang, Fan3 ; Huang, Nisha 1,2; Huang, Haibin4 ; Ma, Chongyang 4; Lee, Tong-Yee 5; Deussen, Oliver 6; Xu, Changsheng1,2
发表期刊	ACM TRANSACTIONS ON GRAPHICS
ISSN	0730-0301
	2023-12-01
卷号	42 期号:6 页码:14
通讯作者	Dong, Weiming(weiming.dong@ia.ac.cn)
摘要	Personalizing generative models offers a way to guide image generation with user-provided references. Current personalization methods can invert an object or concept into the textual conditioning space and compose new natural sentences for text-to-image diffusion models. However, representing and editing specific visual attributes such as material, style, and layout remains a challenge, leading to a lack of disentanglement and editability. To address this problem, we propose a novel approach that leverages the step-by-step generation process of diffusion models, which generate images from low to high frequency information, providing a new perspective on representing, generating, and editing images. We develop the Prompt Spectrum Space P, an expanded textual conditioning space, and a new image representation method called ProSpect. ProSpect represents an image as a collection of inverted textual token embeddings encoded from per-stage prompts, where each prompt corresponds to a specific generation stage (i.e., a group of consecutive steps) of the diffusion model. Experimental results demonstrate that P and ProSpect offer better disentanglement and controllability compared to existing methods. We apply ProSpect in various personalized attribute-aware image generation applications, such as image-guided or text-driven manipulations of materials, style, and layout, achieving previously unattainable results from a single image input without fine-tuning the diffusion models. Our source code is available at https: //github.com/zyxElsa/ProSpect.
关键词	Image generation Diffusion models Attribute-aware editing Model personalization
DOI	10.1145/3618342
收录类别	SCI
语种	英语
资助项目	National Key R&D Program of China[2020AAA0106200] ; National Natural Science Foundation of China[61832016] ; National Natural Science Foundation of China[62102162] ; National Natural Science Foundation of China[U20B2070] ; Beijing Natural Science Foundation[L221013] ; National Science and Technology Council[111-2221-E-006-112-MY3] ; Deutsche Forschungsgemeinschaft (DFG)[413891298]
项目资助者	National Key R&D Program of China ; National Natural Science Foundation of China ; Beijing Natural Science Foundation ; National Science and Technology Council ; Deutsche Forschungsgemeinschaft (DFG)
WOS研究方向	Computer Science
WOS类目	Computer Science, Software Engineering
WOS记录号	WOS:001139790400072
出版者	ASSOC COMPUTING MACHINERY
引用统计	被引频次：1[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/55395
专题	多模态人工智能系统全国重点实验室
通讯作者	Dong, Weiming
作者单位	1.Chinese Acad Sci, Inst Automat, MAIS, Beijing, Peoples R China 2.UCAS, Sch Artificial Intelligence, Beijing, Peoples R China 3.Chinese Acad Sci, Inst Comp Technol, Beijing, Peoples R China 4.Kuaishou Technol, Beijing, Peoples R China 5.Natl Cheng Kung Univ, Tainan, Taiwan 6.Univ Konstanz, Constance, Germany
第一作者单位	中国科学院自动化研究所
通讯作者单位	中国科学院自动化研究所
推荐引用方式 GB/T 7714	Zhang, Yuxin,Dong, Weiming,Tang, Fan,et al. ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models[J]. ACM TRANSACTIONS ON GRAPHICS,2023,42(6):14.
APA	Zhang, Yuxin.,Dong, Weiming.,Tang, Fan.,Huang, Nisha.,Huang, Haibin.,...&Xu, Changsheng.(2023).ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models.ACM TRANSACTIONS ON GRAPHICS,42(6),14.
MLA	Zhang, Yuxin,et al."ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models".ACM TRANSACTIONS ON GRAPHICS 42.6(2023):14.