Pro-tuning: Unified Prompt Tuning for Vision Tasks
Xing Nie1,2; Bolin Ni1,2; Jianlong Chang4; Gaofeng Meng1,2,3; Chunlei Huo1,2; Shiming Xiang1,2; Qi Tian4
Source PublicationIEEE Transactions on Circuits and Systems for Video Technology
2023-10
Volume34Issue:6Pages:4653 - 4667
Abstract

In computer vision, fine-tuning is the de-facto  approach to leverage pre-trained vision models to perform  downstream tasks. However, deploying it in practice is quite  challenging, due to adopting parameter inefficient global update  and heavily relying on high-quality downstream data. Recently,  prompt-based learning, which adds the task-relevant prompt to  adapt the pre-trained models to downstream tasks, has drastically  boosted the performance of many natural language downstream  tasks. In this work, we extend this notable transfer ability  benefited from prompt into vision models as an alternative to  fine-tuning. To this end, we propose parameter-efficient Prompt  tuning (Pro-tuning) to adapt diverse frozen pre-trained models to  a wide variety of downstream vision tasks. The key to Pro-tuning  is prompt-based tuning, i.e., learning task-specific vision prompts  for downstream input images with the pre-trained model frozen. By only training a small number of additional parameters, Protuning  can generate compact and robust downstream models both  for CNN-based and transformer-based network architectures. Comprehensive experiments evidence that the proposed Protuning  outperforms fine-tuning on a broad range of vision  tasks and scenarios, including image classification (under generic  objects, class imbalance, image corruption, adversarial robustness,  and out-of-distribution generalization), and dense prediction  tasks such as object detection and semantic segmentation.

Sub direction classification图像视频处理与分析
planning direction of the national heavy laboratory环境多维感知
Paper associated data
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/57463
Collection多模态人工智能系统全国重点实验室_先进时空数据分析与学习
Affiliation1.the Department of State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Science
2.the School of Artificial Intelligence, University of Chinese Academy of Sciences
3.CAS Centre for Artificial Intelligence and Robotics, HK Institute of Science and Innovation
4.Huawei Cloud & AI
Recommended Citation
GB/T 7714
Xing Nie,Bolin Ni,Jianlong Chang,et al. Pro-tuning: Unified Prompt Tuning for Vision Tasks[J]. IEEE Transactions on Circuits and Systems for Video Technology,2023,34(6):4653 - 4667.
APA Xing Nie.,Bolin Ni.,Jianlong Chang.,Gaofeng Meng.,Chunlei Huo.,...&Qi Tian.(2023).Pro-tuning: Unified Prompt Tuning for Vision Tasks.IEEE Transactions on Circuits and Systems for Video Technology,34(6),4653 - 4667.
MLA Xing Nie,et al."Pro-tuning: Unified Prompt Tuning for Vision Tasks".IEEE Transactions on Circuits and Systems for Video Technology 34.6(2023):4653 - 4667.
Files in This Item: Download All
File Name/Size DocType Version Access License
Pro-tuning_Unified_P(2224KB)期刊论文作者接受稿开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Xing Nie]'s Articles
[Bolin Ni]'s Articles
[Jianlong Chang]'s Articles
Baidu academic
Similar articles in Baidu academic
[Xing Nie]'s Articles
[Bolin Ni]'s Articles
[Jianlong Chang]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Xing Nie]'s Articles
[Bolin Ni]'s Articles
[Jianlong Chang]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: Pro-tuning_Unified_Prompt_Tuning_for_Vision_Tasks.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.