中国科学院自动化研究所机构知识库

Social learning, especially social incentives, is extremely important for humans to achieve a high level of coordination. Inspired by this, we introduce this concept into cooperative multi-agent reinforcement learning (MARL), to implicitly address the credit assignment problem and promote the inter-agent direct interactions for cooperations among agents in cooperative multi-agent games. In this paper, we propose a novel Intrinsic Reward method with Peer Incentives (IRPI) based on actor-critic policy gradient. This method can enable agents to incentivize each other for their cooperations through using causal influence among them. Specifically, a novel intrinsic reward mechanism is innovatively designed to empower each agent the ability to give positive or negative rewards to other peer agents’ actions through considering the causal influence of the other agents on it. The mechanism is realized by a feed-forward neural network through utilizing causal influence between the agents. The causal influence of one agent on another is inferred via counterfactual reasoning using the joint action-value function in MARL. The quality of the influence is assessed via counterfactual reasoning using the individual value function in MARL. Simulations are carried out on two popular multi-agent game testbeds: Starcraft II Micromanagement and Multi-Agent Particle Environments. Simulational results demonstrate that the proposed IRPI can enhance cooperations among the agents to achieve better performance compared with a number of state-of-the-art MARL methods in a variety of cooperative multi-agent games.