中国科学院自动化研究所机构知识库

Swarm systems can cooperatively and efficiently accomplish specified complex tasks. Recent works have shown the potential of multi-agent reinforcement learning methods to study behavior policies of swarm systems. However, it is difficult for them to complete complex swarm tasks efficiently. In human society, role assignment can effectively help humans understand complex tasks and decompose them into simple certain subtasks. Inspired by this, we propose a two-level hierarchical cooperative swarm policy learning framework with role emergence based on hierarchical deep reinforcement learning for distributed swarm systems. In this framework, roles are dynamic and emergent. Agents with the same role tend to collectively complete a certain subtask. Specifically, each agent uses a higher-level swarm policy to dynamically select a role for itself in a role space and at a higher temporal scale, while it uses a lower-level swarm policy to perform the responsibilities of the selected role in a primitive action space. Meanwhile, hierarchical swarm policies with partial observation are centrally trained and decentrally executed, where agents’ local interaction modules and extrinsic team rewards are designed to promote cooperation among agents. In addition, an intrinsic reward is defined to enable different roles to be identified by agents’ longer-term behaviors, which implicitly associates the roles with responsibilities. Simulation results show that our method can learn and generate emergent, dynamic and identifiable roles, which helps swarm systems to reliably and efficiently accomplish complex tasks in a shorter time.