毕业生知识库

最近十年以来，强化学习与深度神经网络广泛结合，并在颇具挑战性的棋牌、即时战略游戏、机器人、自动驾驶等若干领域取得巨大进展。然而，深度强化学习智能体缺乏可解释性和可泛化性，限制了它的进一步应用。一方面，人类用户难以理解智能体的决策依据，从而很难信任与配合智能体。虽然最近可解释人工智能得到了快速发展，但适用于强化学习的解释技术仍然欠缺。另一方面，强化学习中用于训练神经网络的样本包含大量虚假关联，这经常误导智能体在面对未见过的情况时表现不佳。最近因果推理与强化学习的结合逐渐开始得到重视。其中，因果环境模型作为克服强化学习中可解释性与可泛化性挑战的可能途径，表现出强大的潜力。然而，已有研究仍然存在许多缺陷，比如 1）建模需要先验的因果结构知识；2）可解释因果模型的精度往往较差；3）因果关系有时过于密集，导致解释不够简洁和清晰；4）因果解释脱离决策的实际原理，二者缺乏一致性。为了解决上述问题，本文设计了使用因果环境模型解释和指导强化学习的一般算法架构。在这一架构中，更具泛化能力的因果环境模型可被用于指导智能体的决策；同时，模型中的因果知识可被用于构建因果解释链，从而为强化学习提供有效的解释。具体地来说，本文包含了以下两个方面的主要贡献。

一方面，本文探讨了对环境进行因果建模的理论和技术。首先，本文论证了使用特定策略采样数据来学习普遍因果规律的理论可行性。接着，为了克服因果建模的高计算复杂度和低样本效率的问题，本文进一步研究了面向对象的大规模环境中的因果建模。其中，本文假设同类对象在动力学上具有因果对称性。利用这种对称性，本文提出了面向对象的因果动力学模型，它通过面向对象因果图来模块化地描述各类对象的因果关系。本文提出了基于条件互信息的因果发现方法，来识别对象间的模块化因果关系，且这一方法能对环境中不同的对象数目进行泛化。实验结果表明，面向对象的新模型在因果正确率、计算效率、预测精度和泛化能力上，都优于现有的前沿方法。在与规划算法结合来指导决策时，面向对象因果动力学模型的表现也超过了前沿因果模型和主流非因果模型。此外，本文还给出了在因果对称性不满足时建立面向对象模型的方案，这时面向对象因果图仍然作为实际因果关系的有效近似。

另一方面，本文研究了如何建立用于解释的因果环境模型，并让解释与决策在算法架构中被紧密结合起来。本文先后考虑了两种方案：1）动作影响模型方案通过特殊的动作影响预测器，捕捉不同动作下状态变量间的独特因果关系，使得因果结构只反映当前动作下的依赖关系。2）局部因果模型方案将状态和动作都作为约束因果结构的上下文，只在局部上下文中确认变量的相互依赖关系。和用于决策的因果环境模型相比，这两种方案都能产生更简洁的因果规律，具有较好的清晰性和稀疏性。此外，这两种方案都通过特定的转换方法，将解释模型从用于决策的密集模型中导出，进而确保解释和决策本质上都是基于关于环境的一致知识得到的。为了评价解释模型，本文使用因果解释链的客观性质，提出了若干可计算指标，分别评价了 1）因果解释链的可理解性、2）对意图的解释力和 3）对智能体能力的解释力。基于这些指标，本文的实验表明局部因果模型的解释性好于动作影响模型，后者则好于精确的因果环境模型。此外，局部因果模型可以从任何概率型动力学模型中导出，不需要特殊设计的神经网络架构，因此在使用上也更加灵活。

综上所述，本文使用因果推理的相关工具，旨在解决已有研究中结构先验性、低模型精度、高模型密度、模型和解释缺乏一致性等四大问题，最终为克服强化学习中的可解释性和可泛化性两个关键挑战提供可借鉴的方法和思路。

Over the last decade, reinforcement learning has been widely combined with deep neural networks and has achieved great progress in many challenging domains, including board games, real-time strategy games, robotics, autonomous driving, etc. However, agents based on deep reinforcement learning lack explainability and generalizability, which limits their further applications. On one hand, human users have difficulty understanding the decision-making mechanism of the agents, which compromises the trust and cooperation between humans and artificial agents. Although explainable artificial intelligence has developed rapidly in recent years, the explanation techniques for reinforcement learning are still wanting. On the other hand, the samples used to train the neural networks in reinforcement learning contain spurious correlations, which often mislead the agents to perform poorly in unseen situations. Recently, the integration of causal inference and reinforcement learning has begun to attract attention. Among such studies, causal world models have shown great potential as a pathway to overcome the challenges of explainability and generalizability in reinforcement learning. However, existing studies still suffer from several drawbacks: 1) Modeling requires prior causal structure knowledge; 2) Accuracy of explanatory causal models is typically low. 3) Causal relationships are sometimes overly dense, resulting in explanations that are verbose and not perspicuous. 4) Lacking intrinsic consistency, causal explanations are detached from the actual rationale of decision-making. To address the above issues, this study designs a general framework for explaining and guiding reinforcement learning using causal world models. In this framework, a causal world model with improved generalization ability can be used to guide the decision-making of the agent; meanwhile, the causal knowledge in the model can be used to construct explanatory causal chains, which provide effective explanations for the agent's decisions. Specifically, this research contains two major contributions as follows.

On the one hand, this research explores the theories and techniques for causal modeling of environments. First, this study demonstrates the feasibility of using a specific policy to sample data for identifying universal causal relationships. Then, in order to overcome the high computational complexity and low sample efficiency of causal modeling, this research further investigates causal modeling in large-scale object-oriented environments, where the assumption of causal symmetry holds among similar objects. Taking advantage of the symmetry, this study proposes the object-oriented causal dynamics model, which modularly describes the causal relationships of various classes of objects through an object-oriented causal graph. This study also proposes a causal discovery method based on conditional mutual information to identify modular causal relationships among objects, and this method can be generalized to different numbers of objects in the environment. Experimental results show that the proposed object-oriented model outperforms existing state-of-the-art methods in terms of causal correctness, computational efficiency, prediction accuracy, and generalization ability. The object-oriented causal dynamics model also outperforms both other causal models and mainstream non-causal models when combined with planning algorithms to guide decision-making. In addition, this research provides a solution for building object-oriented models with causal symmetry violated, where the object-oriented causal graph still serves as a valid approximation of the actual causal relationships.

On the other hand, this thesis investigates how to create causal world models for explanations, which are tightly integrated with decision-making in the overall framework. Here, two schemes are successively considered: 1) the action-influence modeling scheme captures the unique causal relationships among state variables under different actions through a particular action-influence predictor so that the causal structure reflects only the dependencies under the current action. 2) the local causal modeling scheme treats both the state and the action as the context that constrains the causal structure and identifies dependencies among variables only in the local context. Compared with the causal world models for decision-making, both schemes produce more concise causal connections with better clarity and sparsity. In addition, both schemes include a transition approach that derives the explanatory model from the dense model used by decision-making, ensuring that the explanation and decision-making are essentially based on consistent knowledge about the world. To evaluate the explanatory models, this study proposes several computable criteria according to several objective properties of the explanatory causal chains, that evaluate 1) the interpretability of the causal chains, 2) the explainability for the agent's intentions, and 3) the explainability for the agent's capabilities. Based on these criteria, the experiments in this study show that local causal models have better explanatory power than action-influence models, which in turn are better than the exact causal world models for decision-making. In addition, local causal models are more flexible in use, as they can be derived from any probabilistic dynamics model and do not require a specially designed network architecture.

In summary, using the relevant tools for causal inference, this study aims to address the four major issues of existing studies about causal world models in terms of a priori structures, low model accuracy, high model density, and lacking consistency between the model and explanation. Ultimately, this research would provide valuable methods and ideas to overcome the two key challenges of interpretability and generalizability in reinforcement learning.