FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation
Jie Qin1,2,3; Jie Wu2; Pengxiang Yan2; Ming Li2; Ren Yuxi2; Xuefeng Xiao2; Yitong Wang2; Rui Wang2; Shilei Wen2; Xin Pan2; Xingang Wang1
Conference NameIEEE Conference on Computer Vision and Pattern Recognition
Conference Date6.18-6.22
Conference Place加拿大温哥华市

Recently, open-vocabulary learning has emerged to accomplish segmentation for arbitrary categories of text-based descriptions,  which popularizes the segmentation system to more general-purpose application scenarios. However, existing methods devote to designing specialized architectures or parameters for specific segmentation tasks. These customized design paradigms lead to fragmentation between various segmentation tasks, thus hindering the uniformity of segmentation models. Hence in this paper, we propose FreeSeg, a generic framework to accomplish Unified, Universal and Open-Vocabulary Image Segmentation. FreeSeg optimizes an all-in-one network via one-shot training and employs the same architecture and parameters to handle diverse segmentation tasks seamlessly in the inference procedure.
Additionally, adaptive prompt learning facilitates the unified model to capture task-aware and category-sensitive concepts, improving model robustness in multi-task and varied scenarios. Extensive experimental results demonstrate that FreeSeg establishes new state-of-the-art results in performance and generalization on three segmentation tasks, which outperforms the best task-specific architectures by a large margin: 5.5% mIoU on semantic segmentation, 17.6% mAP on instance segmentation, 20.1% PQ on panoptic segmentation for the unseen class on COCO.

Sub direction classification图像视频处理与分析
planning direction of the national heavy laboratory视觉信息处理
Paper associated data
Document Type会议论文
Affiliation1.Institute of Automation, Chinese Academy of Sciences
2.ByteDance Inc
3.School of Artificial Intelligence, University of Chinese Academy of Sciences
First Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
Jie Qin,Jie Wu,Pengxiang Yan,et al. FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation[C],2023.
Files in This Item: Download All
File Name/Size DocType Version Access License
FreeSeg_Unified Univ(5688KB)会议论文 开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Jie Qin]'s Articles
[Jie Wu]'s Articles
[Pengxiang Yan]'s Articles
Baidu academic
Similar articles in Baidu academic
[Jie Qin]'s Articles
[Jie Wu]'s Articles
[Pengxiang Yan]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Jie Qin]'s Articles
[Jie Wu]'s Articles
[Pengxiang Yan]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: FreeSeg_Unified Universal and Open-Vocabulary Image Segmentation.pdf
Format: Adobe PDF
All comments (0)
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.