Improving Deep Neural Networks Using Softplus Units

	Improving Deep Neural Networks Using Softplus Units
	Hao Zheng1 ; Zhanlei Yang1 ; Wenju Liu1 ; Jizhong Liang 2; Yanpeng Li 2
	2015
会议名称	IJCNN
会议录名称	IJCNN
会议日期	2015
会议地点	Ireland
摘要	Recently, DNNs have achieved great improvement for acoustic modeling in speech recognition tasks. However, it is difficult to train the models well when the depth grows. One main reason is that when training DNNs with traditional sigmoid units, the derivatives damp sharply while back-propagating between layers, which restrict the depth of model especially with insufficient training data. To deal with this problem, some unbounded activation functions have been proposed to preserve sufficient gradients, including ReLU and softplus. Compared with ReLU, the smoothing and nonzero properties of the in gradient makes softplus-based DNNs perform better in both stabilization and performance. However, softplus-based DNNs have been rarely exploited for the phoneme recognition task. In this paper, we explore the use of softplus units for DNNs in acoustic modeling for context-independent phoneme recognition tasks.The revised RBM pre-training and dropout strategy are also applied to improve the performance of softplus units. Experiments show that, the DNNs with softplus units get significantly performance improvement and uses less epochs to get convergence compared to the DNNs trained with standard sigmoid units and ReLUs.
关键词	Softplus Dropout Deep Neural Networks Timit
收录类别	EI
文献类型	会议论文
条目标识符	http://ir.ia.ac.cn/handle/173211/11777
专题	多模态人工智能系统全国重点实验室_机器人视觉
通讯作者	Hao Zheng
作者单位	1.National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences 2.Electric Power Research Institute of Shanxi Electric Power Company
第一作者单位	模式识别国家重点实验室
通讯作者单位	模式识别国家重点实验室
推荐引用方式 GB/T 7714	Hao Zheng,Zhanlei Yang,Wenju Liu,et al. Improving Deep Neural Networks Using Softplus Units[C],2015.

条目包含的文件		下载所有文件
文件名称/大小	文献类型	版本类型	开放类型	使用许可
IJCNN-2015-1.pdf（232KB）	会议论文		开放获取	CC BY-NC-SA	浏览下载