毕业生知识库

心理揣测是能够推断他人心理状态的高级认知功能，为机器赋予心理揣测对于发展高级的人工社会智能起着至关重要的作用。近年来，随着认知神经科学领域对心理揣测的深入研究，其神经机制已被逐步揭示。基于这些研究成果，本文展开对心理揣测计算模型的系统研究。早期的心理揣测计算模型通过概率模型来模拟智能体的归因过程。 目前基于深度学习的心理揣测方法得到了进一步发展，但在生物可解释性方面存在不足。本文受生物脑结构和功能的启发，系统地构建了一个完整的类脑心理揣测脉冲神经网络架构，并使得该模型兼顾生物可解释性与高性能。

具体地，本文基于生物脑中与心理揣测相关的脑区功能及连接，构建了层次化的多脑区协同的心理揣测脉冲神经网络模型，并用其解决简单的单智能体错误信念任务。在该模型的基础上，本文精细化建模了区分自我和他人这项功能，构建了基于自身经验以及对他人观测来揣测他人的模型，并使得具有心理揣测模型的多智能体能够完成合作和竞争任务。进一步，在集体智能中，为了避免为每个智能体单独构建心理揣测模型的计算开销，本文利用在合作任务中智能体主动分享信息的特点，为集体构建了统一的心理揣测模型。最后，本文针对现有模型仅依赖历史信息对他人进行揣测的问题，进一步地提出了一阶心理揣测模型，实现了对他人更深层次的推理，赋予人工智能系统更为深入的社会认知能力。本文具体贡献将从以下四个方面进行介绍。
一、 多脑区协同的类脑心理揣测脉冲神经网络模型。 本文提出了一个多脑区协同的类脑心理揣测脉冲神经网络模型，旨在借鉴心理揣测能力解决人工智能安全风险问题。 该模型依托于认知神经科学的研究内容，用脉冲神经网络分别模拟了颞顶交界处、前额叶皮层、前扣带皮层和额下回等脑区的特定功能，并构建了这些子网络之间的连接关系。本文采用具有高度生物可解释性的脉冲时序依赖可塑性以及基于多巴胺调控的可塑性机制对网络进行训练。 模型使智能体具备区分自我和他人、识别其他智能体的错误信念、基于自身经验推断他人处境的能力。此外，基于心理学实验范式，本文设计了测试心理揣测能力的实验。实验结果表明，在带有随机起点和障碍物的网格环境中，该模型能够成功辨别他人错误信念、评估他人的安全风险。

二、 多智能体心理揣测脉冲神经网络模型。 在上述研究基础上，本文进一步细化了智能体区分自我和他人、模拟他人决策的功能，构建了多智能体心理揣测脉冲神经网络模型，旨在利用个体的心理揣测能力提升团队合作和竞争的效率。该模型精细化地构建了基于自我经验和对他人观测来揣测他人的行为的模块，并以此来模拟颞顶交界处（区分自我和他人信息）以及前额叶皮层（基于两种不同信息推测他人）的脑区功能。本文将多智能体心理揣测脉冲神经网络模型分别集成到基于值和基于策略的多智能体强化学习算法中，使智能体依据对他人行动的预测进行自适应行为调整。实验结果证明，整合心理揣测的决策模型在合作和竞争任务中的性能明显超过强化学习算法基线。
三、 集体心理揣测脉冲神经网络模型。 在合作任务中，为了避免为每个智能体单独建立一个心理揣测网络所带来的计算开销，本文借鉴认知科学中集体心理揣测的概念，即多智能体在集体中共享信息特点，提出了集体心理揣测脉冲神经网络模型。该模型构建了由观测信息到集体心理状态的编码器，以及由集体心理状态到预测信息的解码器，此过程受最小化自由能的约束。集体决策模型与集体心理揣测模型之间的信息交换能够模拟集体与外部环境的互动，进而促进策略训练，提升学习效率及样本利用率。在合作型多智能体开源环境（星际争霸、多智能体粒子环境）的实验中，本文提出的模型相较于多个多智能体强化学习算法基线，在提升合作效率和性能方面表现卓越，并展现了良好的迁移能力。

四、 一阶心理揣测脉冲神经网络模型。 本文在推断他人心理状态的基础上，进一步受到一阶心理揣测的启发，提出了一种基于递归推理的一阶心理揣测脉冲神经网络模型，旨在实现对他人深层次推理。该模型基于自监督训练方法将一阶心理揣测与零阶心理揣测融合，得到对他人行为的综合推理。此外，本文设计了一种基于期望对齐的自组织内在奖励更新算法，使得智能体根据对他人的递归推理实时调整内在奖励以灵活应对他人的策略。在多个合作和竞争的实验中，相比于多智能体强化学习算法以及零阶心理揣测模型，一阶心理揣测模型性能显著提升。

综上所述，本文由浅入深、由宽泛到精细，逐步构建了层次化的多脑区协同的心理揣测模型、能够用自身经验或者对他人观测来揣测合作者和竞争者的多智能体心理揣测模型、通过共享信息减少计算复杂度的集体心理揣测模型以及涉及递归推理的一阶心理揣测模型。实验结果证明，所提类脑心理揣测脉冲神经网络模型整体上达到了可解释性与性能兼备的目标。

Theory of Mind (ToM) is a high-level cognitive function that allows for the inference of others’ mental states. Equipping machines with theory of mind is crucial for the development of advanced artificial social intelligence. In recent years, with the deepening research in the field of cognitive neuroscience on theory of mind, its neural mechanisms have been gradually unveiled. Based on these research findings, this paper conducts a systematic study of computational models for theory of mind. Early computational models of theory of mind attempted to simulate the attribution process of agents through probabilistic models. Currently, theory of mind models based on deep learning have seen further development, yet they lack in terms of biological interpretability. Inspired by the structure and function of the brain, this paper systematically constructs a biologically interpretable spiking neural network architecture for brain-inspired theory of mind and ensures that the model balances biological interpretability with high performance.

Specifically, this paper constructs a hierarchical multi-brain area coordinated theory of mind spiking neural network model based on the brain area functions and connections related to theory of mind in the biological brain, and uses it to solve simple single-agent false belief tasks. Based on this model, this paper finely models the function of distinguishing self from others, constructs a model of speculating on others based on self-experience and observation of others, and enables multi-agents with theory of mind models to complete cooperation and competition tasks. Furthermore, to avoid the computational cost of building a theory of mind model for each agent separately, this paper utilizes the characteristic of agents actively sharing information in cooperative tasks to construct a unified theory of mind model for the collective. Finally, addressing
the issue that existing models only rely on historical information to speculate on others, this paper further proposes a first-order theory of mind model, achieving deeper inference on others and endowing artificial intelligence systems with more profound social cognitive abilities. The specific contributions of this paper will be elaborated from the following four aspects.

1. A brain-inspired theory of mind spiking neural network model. This paper
presents a multi-brain area coordinated theory of mind spiking neural network model, aimed at leveraging the ability of theory of mind to address safety risks in artificial intelligence. The model draws upon the findings of cognitive neuroscience research, using spiking neural networks to simulate the specific functions of brain areas such as the temporo-parietal junction, the prefrontal cortex, the anterior cingulate cortex, and the inferior frontal gyrus, and establishes the interconnections between these subnetworks. The network is trained using spike-timing-dependent plasticity, which has a high degree of biological interpretability, as well as plasticity mechanisms modulated by dopamine. The model endows agents with the capability to distinguish between self and others, recognize the false beliefs of other agents, and infer the situations of others based on their own experiences. Additionally, based on theory of mind paradigms, this paper designs experiments to test the ability of theory of mind. The experimental results demonstrate that in a grid environment with random starting points and obstacles, the model can successfully identify others’ false beliefs and assess their safety risks.

2. A multi-agnet theory of mind spiking neural network model. Building upon
the aforementioned research, this paper further refines the functionalities of agents to distinguish between self and others, and to simulate the decision-making of others. We have constructed a multi-agent theory of mind spiking neural network model, aimed at enhancing the efficiency of teamwork and competition through individual theory of mind capabilities. The model constructs modules that infer the behavior of others based on self-experience and observations of others, thereby simulating the functions of brain areas such as the temporo-parietal junction, which distinguishes between self and other, and the prefrontal cortex, which infers others based on two distinct types of information. This paper integrates the multi-agent theory of mind spiking neural network model into both value-based and policy-based multi-agent reinforcement learning algorithms, enabling agents to adaptively adjust their behavior based on predictions of others’ actions. Experimental results substantiate that the decision-making model integrated with ToM significantly outperforms baselines in cooperative and competitive tasks. 

3. A theory of collective mind spiking neural network model. In the context of
cooperative tasks, to mitigate the computational expense incurred by establishing a separate theory of mind network for each agent, this paper draws on the concept of theory of collective mind from cognitive science, which involves the sharing of informational characteristics among a group of agents. We propose a theory of collective mind spiking neural network model that constructs an encoder from observational information to collective mental states and a decoder from collective mental states to predictive information, a process constrained by the minimization of free energy. The exchange of information between the collective decision-making model and the theory of collective
mind model can simulate the interaction of the collective with the external environment, thereby enhancing learning efficiency and sample utilization. In experiments within cooperative multi-agent open-source environments (such as StarCraft and the MultiAgent Particle Environment), the proposed model demonstrated superior performance over several multi-agent reinforcement learning algorithm baselines in improving cooperative efficiency and performance, and also exhibited commendable transferability.

4. A first-order theory of mind spiking neural network model. Building upon
inferring others’ mental states based on their behavior, this paper further proposes a first-order theory of mind spiking neural network model inspired by first-order theory of mind, aiming to achieve deeper inference based on others’ mental states. This model integrates first-order theory of mind with zero-order theory of mind through a self-supervised training method, resulting in comprehensive reasoning about others’ behaviors. Additionally, this study designed a self-organizing intrinsic reward update algorithm based on expectation alignment, enabling agents to adjust their intrinsic rewards in real-time based on recursive reasoning about others, thus flexibly responding
to others’ strategies. Experimental results in various mixed cooperation and competition scenarios (multi-agent particle environments) demonstrate that agents equipped with the first-order theory of mind model significantly improve performance compared to multi-agent reinforcement learning algorithms and zero-order theory of mind models.

In summary, this paper has progressively constructed a hierarchical model of theory of mind that operates through coordinated brain areas, starting from a foundational understanding and moving towards a refined model. It has developed a multi-agent theory of mind model capable of inferring the mental states of collaborators and competitors using self-experience or observations of others. Additionally, it has introduced a theory of collective mind model that reduces computational complexity through information sharing, as well as a first-order ToM model involving recursive reasoning. Experimental results have confirmed that the proposed brain-inspired theory of mind spiking neural network models have achieved the dual goals of biological interpretability and performance.