CASIA OpenIR  > 智能感知与计算
Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
Peng HW(彭厚文)1; Du, Hao2; Yu HY(俞宏远)3; Li, Qi4; Liao, Jing2; Fu, Jianlong1
Conference NameConference on Neural Information Processing Systems
Conference Date2020
Conference PlaceVancouver, Canada
Publication PlaceNeurIPS

One-shot weight sharing methods have recently drawn great attention in neural architecture search due to high efficiency and competitive performance. However, weight sharing across models has an inherent deficiency, i.e., insufficient training of subnetworks in hypernetworks. To alleviate this problem, we present a simple yet effective architecture distillation method. The central idea is that subnetworks can learn collaboratively and teach each other throughout the training process, aiming to boost the convergence of individual models. We introduce the concept of prioritized path, which refers to the architecture candidates exhibiting superior performance during training. Distilling knowledge from the prioritized paths is able to boost the training of subnetworks. Since the prioritized paths are changed on the fly depending on their performance and complexity, the final obtained paths are the cream of the crop. We directly select the most promising one from the prioritized paths as the final architecture, without using other complex search methods, such as reinforcement learning or evolution algorithms. The experiments on ImageNet verify such path distillation method can improve the convergence ratio and performance of the hypernetwork, as well as boosting the training of subnetworks. The discovered architectures achieve superior performance compared to the recent MobileNetV3 and EfficientNet families under aligned settings. Moreover, the experiments on object detection and more challenging search space show the generality and robustness of the proposed method. Code and models are available at

Document Type会议论文
Corresponding AuthorPeng HW(彭厚文)
Affiliation1.Microsoft Research Asia
2.City University of Hong Kong
3.Chinese Academy of Sciences
4.Tsinghua University
Recommended Citation
GB/T 7714
Peng HW,Du, Hao,Yu HY,et al. Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search[C]. NeurIPS:NeurIPS,2020.
Files in This Item:
File Name/Size DocType Version Access License
NeurIPS-2020-cream-o(357KB)会议论文 开放获取CC BY-NC-SAView
Related Services
Recommend this item
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Peng HW(彭厚文)]'s Articles
[Du, Hao]'s Articles
[Yu HY(俞宏远)]'s Articles
Baidu academic
Similar articles in Baidu academic
[Peng HW(彭厚文)]'s Articles
[Du, Hao]'s Articles
[Yu HY(俞宏远)]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Peng HW(彭厚文)]'s Articles
[Du, Hao]'s Articles
[Yu HY(俞宏远)]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: NeurIPS-2020-cream-of-the-crop-distilling-prioritized-paths-for-one-shot-neural-architecture-search-Paper.pdf
Format: Adobe PDF
All comments (0)
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.