CASIA OpenIR
Primal Averaging: A New Gradient Evaluation Step to Attain the Optimal Individual Convergence
Tao, Wei1; Pan, Zhisong1; Wu, Gaowei2; Tao, Qing2,3
Source PublicationIEEE TRANSACTIONS ON CYBERNETICS
ISSN2168-2267
2020-02-01
Volume50Issue:2Pages:835-845
Corresponding AuthorTao, Qing(qing.tao@ia.ac.cn)
AbstractMany well-known first-order gradient methods have been extended to cope with large-scale composite problems, which often arise as a regularized empirical risk minimization in machine learning. However, their optimal convergence is attained only in terms of the weighted average of past iterative solutions. How to make the individual convergence of stochastic gradient descent (SGD) optimal, especially for strongly convex problems has now become a challenging problem in the machine learning community. On the other hand, Nesterov's recent weighted averaging strategy succeeds in achieving the optimal individual convergence of dual averaging (DA) but it fails in the basic mirror descent (MD). In this paper, a new primal averaging (PA) gradient operation step is presented, in which the gradient evaluation is imposed on the weighted average of all past iterative solutions. We prove that simply modifying the gradient operation step in MD by PA strategy suffices to recover the optimal individual rate for general convex problems. Along this line, the optimal individual rate of convergence for strongly convex problems can also be achieved by imposing the strong convexity on the gradient operation step. Furthermore, we extend PA-MD to solve regularized nonsmooth learning problems in the stochastic setting, which reveals that PA strategy is a simple yet effective extra step toward the optimal individual convergence of SGD. Several real experiments on sparse learning and SVM problems verify the correctness of our theoretical analysis.
KeywordConvergence Convex functions Machine learning Optimization methods Linear programming Cybernetics Individual convergence machine learning mirror descent (MD) methods regularized learning problems stochastic gradient descent (SGD) stochastic optimization
DOI10.1109/TCYB.2018.2874332
WOS KeywordNEURAL-NETWORK ; OPTIMIZATION ; PERFORMANCE ; ALGORITHMS
Indexed BySCI
Language英语
Funding ProjectNSFC[61673394]
Funding OrganizationNSFC
WOS Research AreaAutomation & Control Systems ; Computer Science
WOS SubjectAutomation & Control Systems ; Computer Science, Artificial Intelligence ; Computer Science, Cybernetics
WOS IDWOS:000506849800036
PublisherIEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
Citation statistics
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/29522
Collection中国科学院自动化研究所
Corresponding AuthorTao, Qing
Affiliation1.Army Engn Univ PLA, Command & Control Engn Coll, Nanjing 210007, Peoples R China
2.Chinese Acad Sci, Inst Automat, Beijing 100190, Peoples R China
3.Army Acad Artillery & Air Def, Dept Comp Sci, Hefei 230031, Peoples R China
Corresponding Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
Tao, Wei,Pan, Zhisong,Wu, Gaowei,et al. Primal Averaging: A New Gradient Evaluation Step to Attain the Optimal Individual Convergence[J]. IEEE TRANSACTIONS ON CYBERNETICS,2020,50(2):835-845.
APA Tao, Wei,Pan, Zhisong,Wu, Gaowei,&Tao, Qing.(2020).Primal Averaging: A New Gradient Evaluation Step to Attain the Optimal Individual Convergence.IEEE TRANSACTIONS ON CYBERNETICS,50(2),835-845.
MLA Tao, Wei,et al."Primal Averaging: A New Gradient Evaluation Step to Attain the Optimal Individual Convergence".IEEE TRANSACTIONS ON CYBERNETICS 50.2(2020):835-845.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Tao, Wei]'s Articles
[Pan, Zhisong]'s Articles
[Wu, Gaowei]'s Articles
Baidu academic
Similar articles in Baidu academic
[Tao, Wei]'s Articles
[Pan, Zhisong]'s Articles
[Wu, Gaowei]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Tao, Wei]'s Articles
[Pan, Zhisong]'s Articles
[Wu, Gaowei]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.