CASIA OpenIR  > 模式识别国家重点实验室  > 机器人视觉
Improving Deep Neural Networks Using Softplus Units
Hao Zheng1; Zhanlei Yang1; Wenju Liu1; Jizhong Liang2; Yanpeng Li2
Conference NameIJCNN
Source PublicationIJCNN
Conference Date2015
Conference PlaceIreland
AbstractRecently, DNNs have achieved great improvement for acoustic modeling in speech recognition tasks. However, it is difficult to train the models well when the depth grows. One main reason is that when training DNNs with traditional sigmoid units, the derivatives damp sharply while back-propagating between layers, which restrict the depth of model especially with insufficient training data. To deal with this problem, some unbounded activation functions have been proposed to preserve sufficient gradients, including ReLU and softplus. Compared with ReLU, the smoothing and nonzero properties of the in gradient makes softplus-based DNNs perform better in both stabilization and performance. However, softplus-based DNNs have been rarely
exploited for the phoneme recognition task. In this paper, we explore the use of softplus units for DNNs in acoustic modeling for context-independent phoneme recognition tasks.The revised RBM pre-training and dropout strategy are also applied to improve the performance of softplus units. Experiments show that, the DNNs with softplus units get significantly performance improvement and uses less epochs to get convergence compared to the DNNs trained with standard sigmoid units and ReLUs.
KeywordSoftplus Dropout Deep Neural Networks Timit
Indexed ByEI
Document Type会议论文
Corresponding AuthorHao Zheng
Affiliation1.National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences
2.Electric Power Research Institute of Shanxi Electric Power Company
Recommended Citation
GB/T 7714
Hao Zheng,Zhanlei Yang,Wenju Liu,et al. Improving Deep Neural Networks Using Softplus Units[C],2015.
Files in This Item: Download All
File Name/Size DocType Version Access License
IJCNN-2015-1.pdf(232KB)会议论文 开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Hao Zheng]'s Articles
[Zhanlei Yang]'s Articles
[Wenju Liu]'s Articles
Baidu academic
Similar articles in Baidu academic
[Hao Zheng]'s Articles
[Zhanlei Yang]'s Articles
[Wenju Liu]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Hao Zheng]'s Articles
[Zhanlei Yang]'s Articles
[Wenju Liu]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: IJCNN-2015-1.pdf
Format: Adobe PDF
All comments (0)
No comment.

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.