Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search | |
Peng HW(彭厚文)1; Du, Hao2; Yu HY(俞宏远)3; Li, Qi4; Liao, Jing2; Fu, Jianlong1 | |
2020 | |
会议名称 | Conference on Neural Information Processing Systems |
会议日期 | 2020 |
会议地点 | Vancouver, Canada |
出版地 | NeurIPS |
出版者 | NeurIPS |
摘要 | One-shot weight sharing methods have recently drawn great attention in neural architecture search due to high efficiency and competitive performance. However, weight sharing across models has an inherent deficiency, i.e., insufficient training of subnetworks in hypernetworks. To alleviate this problem, we present a simple yet effective architecture distillation method. The central idea is that subnetworks can learn collaboratively and teach each other throughout the training process, aiming to boost the convergence of individual models. We introduce the concept of prioritized path, which refers to the architecture candidates exhibiting superior performance during training. Distilling knowledge from the prioritized paths is able to boost the training of subnetworks. Since the prioritized paths are changed on the fly depending on their performance and complexity, the final obtained paths are the cream of the crop. We directly select the most promising one from the prioritized paths as the final architecture, without using other complex search methods, such as reinforcement learning or evolution algorithms. The experiments on ImageNet verify such path distillation method can improve the convergence ratio and performance of the hypernetwork, as well as boosting the training of subnetworks. The discovered architectures achieve superior performance compared to the recent MobileNetV3 and EfficientNet families under aligned settings. Moreover, the experiments on object detection and more challenging search space show the generality and robustness of the proposed method. Code and models are available at https://github.com/microsoft/cream.git. |
文献类型 | 会议论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/48710 |
专题 | 智能感知与计算研究中心 |
通讯作者 | Peng HW(彭厚文) |
作者单位 | 1.Microsoft Research Asia 2.City University of Hong Kong 3.Chinese Academy of Sciences 4.Tsinghua University |
推荐引用方式 GB/T 7714 | Peng HW,Du, Hao,Yu HY,et al. Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search[C]. NeurIPS:NeurIPS,2020. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
NeurIPS-2020-cream-o(357KB) | 会议论文 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论