Institutional Repository of Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China
Collaborative Deconvolutional Neural Networks for Joint Depth Estimation and Semantic Segmentation | |
Liu, Jing1; Wang, Yuhang1,2; Li, Yong1,2; Fu, Jun1,2; Li, Jiangyun3; Lu, Hanqing1 | |
发表期刊 | IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS |
ISSN | 2162-237X |
2018-11-01 | |
卷号 | 29期号:11页码:5655-5666 |
通讯作者 | Liu, Jing(jliu@nlpr.ia.ac.cn) |
摘要 | Semantic segmentation and single-view depth estimation are two fundamental problems in computer vision. They exploit the semantic and geometric properties of images, respectively, and are thus complementary in scene understanding. In this paper, we propose a collaborative deconvolutional neural network (C-DCNN) to jointly model these two problems for mutual promotion. The C-DCNN consists of two DCNNs, of which each is for one task. The DCNNs provide a finer resolution reconstruction method and are pretrained with hierarchical supervision. The feature maps from these two DCNNs are integrated via a pointwise bilinear layer, which fuses the semantic and depth information and produces higher order features. Then, the integrated features are fed into two sibling classification layers to simultaneously learn for semantic segmentation and depth estimation. In this way, we combine the semantic and depth features in a unified deep network and jointly train them to benefit each other. Specifically, during network training, we process depth estimation as a classification problem where a soft mapping strategy is proposed to map the continuous depth values into discrete probability distributions and the cross entropy loss is used. Besides, a fully connected conditional random field is also used as postprocessing to further improve the performance of semantic segmentation, where the proximity relations of pixels on position, intensity, and depth are jointly considered. We evaluate our approach on two challenging benchmarks: NYU Depth V2 and SUN RGB-D. It is demonstrated that our approach effectively utilizes these two kinds of information and achieves state-of-the-art results on both the semantic segmentation and depth estimation tasks. |
关键词 | Deconvolutional neural network (DCNN) depth estimation fully connected conditional random field (CRF) pointwise bilinear layer semantic segmentation soft mapping strategy |
DOI | 10.1109/TNNLS.2017.2787781 |
关键词[WOS] | RECOGNITION |
收录类别 | SCI |
语种 | 英语 |
资助项目 | National Natural Science Foundation of China[61332016] ; National Natural Science Foundation of China[61472422] ; Fundamental Research Funds for Central Universities[FRFBD-16-005A] ; National Natural Science Foundation of China[61332016] ; National Natural Science Foundation of China[61472422] ; Fundamental Research Funds for Central Universities[FRFBD-16-005A] |
项目资助者 | National Natural Science Foundation of China ; Fundamental Research Funds for Central Universities |
WOS研究方向 | Computer Science ; Engineering |
WOS类目 | Computer Science, Artificial Intelligence ; Computer Science, Hardware & Architecture ; Computer Science, Theory & Methods ; Engineering, Electrical & Electronic |
WOS记录号 | WOS:000447832200038 |
出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/28102 |
专题 | 模式识别国家重点实验室_图像与视频分析 |
通讯作者 | Liu, Jing |
作者单位 | 1.Chinese Acad Sci, Inst Automat, Natl Lab Pattern Recognit, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 3.Univ Sci & Technol, Beijing 100083, Peoples R China |
第一作者单位 | 模式识别国家重点实验室 |
通讯作者单位 | 模式识别国家重点实验室 |
推荐引用方式 GB/T 7714 | Liu, Jing,Wang, Yuhang,Li, Yong,et al. Collaborative Deconvolutional Neural Networks for Joint Depth Estimation and Semantic Segmentation[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2018,29(11):5655-5666. |
APA | Liu, Jing,Wang, Yuhang,Li, Yong,Fu, Jun,Li, Jiangyun,&Lu, Hanqing.(2018).Collaborative Deconvolutional Neural Networks for Joint Depth Estimation and Semantic Segmentation.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,29(11),5655-5666. |
MLA | Liu, Jing,et al."Collaborative Deconvolutional Neural Networks for Joint Depth Estimation and Semantic Segmentation".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 29.11(2018):5655-5666. |
条目包含的文件 | 条目无相关文件。 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论