Knowledge Commons of Institute of Automation,CAS
A Novel Heterogeneous Actor-critic Algorithm with Recent Emphasizing Replay Memory | |
Bao Xi1,2![]() ![]() ![]() ![]() | |
发表期刊 | International Journal of Automation and Computing
![]() |
ISSN | 1476-8186 |
2021 | |
卷号 | 18期号:4页码:619-631 |
摘要 | Reinforcement learning (RL) algorithms have been demonstrated to solve a variety of continuous control tasks. However, the training efficiency and performance of such methods limit further applications. In this paper, we propose an off-policy heterogeneous actor-critic (HAC) algorithm, which contains soft Q-function and ordinary Q-function. The soft Q-function encourages the exploration of a Gaussian policy, and the ordinary Q-function optimizes the mean of the Gaussian policy to improve the training efficiency. Experience replay memory is another vital component of off-policy RL methods. We propose a new sampling technique that emphasizes recently experienced transitions to boost the policy training. Besides, we integrate HAC with hindsight experience replay (HER) to deal with sparse reward tasks, which are common in the robotic manipulation domain. Finally, we evaluate our methods on a series of continuous control benchmark tasks and robotic manipulation tasks. The experimental results show that our method outperforms prior state-of-the-art methods in terms of training efficiency and performance, which validates the effectiveness of our method. |
关键词 | Reinforcement learning (RL) actor-critic experience replay training efficiency manipulation skill learning |
DOI | 10.1007/s11633-021-1296-x |
七大方向——子方向分类 | 其他 |
国重实验室规划方向分类 | 其他 |
是否有论文关联数据集需要存交 | 否 |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/45068 |
专题 | 学术期刊_Machine Intelligence Research |
作者单位 | 1.State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China 2.University of Chinese Academy of Sciences, Beijing 100049, China 3.Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai 200031, China |
第一作者单位 | 中国科学院自动化研究所 |
推荐引用方式 GB/T 7714 | Bao Xi,Rui Wang,Ying-Hao Cai,et al. A Novel Heterogeneous Actor-critic Algorithm with Recent Emphasizing Replay Memory[J]. International Journal of Automation and Computing,2021,18(4):619-631. |
APA | Bao Xi,Rui Wang,Ying-Hao Cai,TaoLu,&Shuo Wang.(2021).A Novel Heterogeneous Actor-critic Algorithm with Recent Emphasizing Replay Memory.International Journal of Automation and Computing,18(4),619-631. |
MLA | Bao Xi,et al."A Novel Heterogeneous Actor-critic Algorithm with Recent Emphasizing Replay Memory".International Journal of Automation and Computing 18.4(2021):619-631. |
条目包含的文件 | ||||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
IJAC-2020-08-215.pdf(2505KB) | 期刊论文 | 出版稿 | 开放获取 | CC BY-NC-SA | 浏览 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论