Towards Better Generalization of Deep Neural Networks via Non-Typicality Sampling Scheme

doi:10.1109/TNNLS.2022.3147031

CASIA OpenIR > 多模态人工智能系统全国重点实验室 > 平行智能技术与系统团队

	Towards Better Generalization of Deep Neural Networks via Non-Typicality Sampling Scheme
	Peng, Xinyu 1; Wang, Fei-Yue2 ; Li, Li 3
发表期刊	IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
ISSN	2162-237X
	2022-02-11
页码	11
通讯作者	Li, Li(li-li@tsinghua.edu.cn)
摘要	Improving the generalization performance of deep neural networks (DNNs) trained by minibatch stochastic gradient descent (SGD) has raised lots of concerns from deep learning practitioners. The standard simple random sampling (SRS) scheme used in minibatch SGD treats all training samples equally in gradient estimation. In this article, we study a new data selection method based on the intrinsic property of the training set to help DNNs have better generalization performance. Our theoretical analysis suggests that this new sampling scheme, called the nontypicality sampling scheme, boosts the generalization performance of DNNs through biasing the solution toward wider minima, under certain assumptions. We confirm our findings experimentally and show that more variants of minibatch SGD can also benefit from the new sampling scheme. Finally, we discuss an extension of the nontypicality sampling scheme that holds promise to enhance both generalization performance and convergence speed of minibatch SGD.
关键词	Training Estimation Deep learning Standards Optimization Noise measurement Convergence Deep learning generalization performance nontypicality sampling scheme stochastic gradient descent (SGD)
DOI	10.1109/TNNLS.2022.3147031
收录类别	SCI
语种	英语
资助项目	National Key Research and Development Program of China[2020AAA0108104]
项目资助者	National Key Research and Development Program of China
WOS研究方向	Computer Science ; Engineering
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Hardware & Architecture ; Computer Science, Theory & Methods ; Engineering, Electrical & Electronic
WOS记录号	WOS:000757938800001
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：1[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/47889
专题	多模态人工智能系统全国重点实验室_平行智能技术与系统团队
通讯作者	Li, Li
作者单位	1.Tsinghua Univ, Dept Automat, Beijing 100084, Peoples R China 2.Chinese Acad Sci, State Key Lab Management & Control Complex Syst, Inst Automat, Beijing 100080, Peoples R China 3.Tsinghua Univ, Tsinghua Natl Lab Informat Sci & Technol, Dept Automat, Beijing 100084, Peoples R China
推荐引用方式 GB/T 7714	Peng, Xinyu,Wang, Fei-Yue,Li, Li. Towards Better Generalization of Deep Neural Networks via Non-Typicality Sampling Scheme[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2022:11.
APA	Peng, Xinyu,Wang, Fei-Yue,&Li, Li.(2022).Towards Better Generalization of Deep Neural Networks via Non-Typicality Sampling Scheme.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,11.
MLA	Peng, Xinyu,et al."Towards Better Generalization of Deep Neural Networks via Non-Typicality Sampling Scheme".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS (2022):11.