毕业生知识库

人脑对自然语言的理解是复杂的认知过程。大脑接收到语言信号后，需要建立语义信息和句法信息的表征，前者将语言符号与真实世界中的事物和以往经验联系起来，而后者可以将小的语言单位组成大的语言单位。研究大脑的语义和句法表征机制是解析大脑语言功能的关键，对于探索人脑的语言理解机制、建立更加有效的自然语言处理模型有着重要的理论意义和应用价值。

由于语言理解过程中语义和句法的效应混合在一起难以分离，已有研究大多基于控制实验的范式，通过受控语料分离二者的效应。然而受控语料与自然语言差异较大，存在结论难以泛化等问题。

本文在自然语言刺激下研究大脑对语义和句法的表征机制。针对目前缺少高质量的自然语言刺激神经影像数据、语义和句法特征抽取和量化表示困难等问题，本文采集了神经影像数据，并结合自然语言处理中的分布式文本表示与认知科学领域的经验语义表示，通过计算方法分别建模语义和句法特征与大脑活动的映射关系，对大脑的句法和语义表征机制进行了研究。

论文的主要工作和创新点归纳如下：

1. 构建了汉语自然语言刺激的神经影像数据集

大脑理解语言时的活动数据是研究大脑语言处理机制的基础，数据的质量很大程度上会限制研究的结果。自然语言刺激下的研究需要使用计算模型建立语言刺激与脑活动信号的映射关系，对神经影像数据的规模和质量都有较高的要求。针对当前缺少基于汉语自然语言刺激的神经影像数据这一问题，本文采集了12位汉语母语被试收听总时长约5小时的汉语故事音频时的功能磁共振影像数据，对所采集的神经影像数据进行了预处理和技术验证，验证结果表明数据具有较高的质量。此外，本文对刺激故事文本进行了语言特征标注，为句法和语义的大脑表征机制分析奠定了基础。

2. 提出了一种特征消除方法来分析词汇级句法信息的大脑表征机制

词汇是语言中最小的独立运用的音义结合体，是组成复杂短语和句子的基本单位。研究词汇如何组成短语和句子必须先对词进行分类，并且理解词与词之间的关系。因此，本文选择了三种句法特征：词性、依存关系、谓词-论元结构，在汉语和英语两种语言上分析了大脑的词汇句法特征表征模式。为了解决句法特征表示困难且各特征之间以及与语义信息之间难以分离的问题，本文提出了特征消除方法，从分布式文本表示模型计算的词向量中分离出不同句法特征。本文提出的特征消除方法能够将特定特征从词向量空间中消除，同时保留其他特征，从而较好实现特征分离。随后，本文使用体素编码方法分别基于原始词向量和单一特征消除后的词向量预测大脑的活动，通过验证特征消除后预测准确率下降的脑区得到每种句法特征在大脑皮层上的分布。实验表明句法特征在大脑中的表征呈现分布式的特点，揭示了脑区之间精细的句法分离与重合模式，且发现了汉语和英语之间脑区分离与重合模式的差异。

3. 发现了大脑构建层次句法结构时采用适应语言句法结构的计算方式

层次性是语言的本质属性，在句子中，词汇按照一定的句法规则逐层进行组合。本文针对语言的这一特点，研究了汉语和英语中层次句法结构在大脑中的构建方式。为了探究大脑构建句法结构机制的一般规律，本文从汉英两种语言的结构差异出发，量化分析了两种语言的句法结构分支方向差异和句法分析策略所产生的记忆负担之间的关系。分析结果表明汉英的语言结构存在明显差异，英语以右分支为主，而汉语则左右分支都有。这种差异导致不同句法分析策略会产生不同的记忆负担。对于汉语来说，自下而上的方式造成的记忆负担更小，而英语则相反，自上而下的方式记忆负担更小。随后，本文通过神经影像分析发现汉英母语者的脑活动均与记忆负担更小的句法分析策略更符合，表明大脑受到认知资源限制，在构建层次句法结构时采取适应语言句法结构的计算方式。

4. 基于一组具有心智经验基础的语义特征揭示了大脑的语义表征机制

大脑如何表征语义是认知神经科学和心理学等多个学科共同关注的问题。已有研究表明大脑获取语义的过程与心智经验密不可分，包括具体的感觉-运动经验和抽象的情感和社会经验等。但是，已有研究大多仅关注其中的某一类经验语义，忽略另一类经验对语义表征的重要性。本文选择了一组具有代表性的感觉-运动和非感觉-运动语义特征，包括视觉、动作、情绪、社会、空间、时间，通过人工标注的方式为汉语自然语言刺激中的词汇标注了每个维度的分数。然后，本文通过线性模型建立这组语义特征与大脑活动之间的联系，并在此基础上分析不同语义维度在大脑中表征的分离与重合模式。实验结果发现了大脑中多个对语义信息敏感的脑区，并揭示了语义在大脑中复杂的组织分布，为大脑语义系统的组织模式提供了新的见解。

Language understanding in the brain is a complex cognitive process. After receiving language signals, the brain needs to build semantic and syntactic representations. The former associates linguistic symbols with real-world objects, actions, and previous experiences, and the latter enables humans to combine small linguistic units into large linguistic units. Studying the semantic and syntactic representation of the brain is the key to exploring the language understanding mechanism of the human brain and inspiring more effective brain-like natural language processing models. It has important theoretical significance and application value.

Since the semantic and syntactic effects in the process of language understanding are mixed together and difficult to separate, most of the existing studies are based on the paradigm of controlled experiments, and try to separate the effects of the two through the controlled corpus. However, the controlled corpus is quite different from natural language and therefore brings problems such as generalizing conclusions outside controlled conditions.

This thesis studies the brain representation mechanism of semantics and syntax under naturalistic language stimuli. In view of the current lack of high-quality neuroimaging data collected under naturalistic language stimuli and the difficulty of extracting and quantifying semantic and syntactic features, this thesis first collects neuroimaging data. Then, this thesis leverages the distributed text representation in natural language processing and the psychologically plausible semantic representation in the field of cognitive science to represent syntactic and semantic features. With the collected neuroimaging data and the extracted features, this thesis studies how the brain represents syntax and semantics by modeling the mapping relationship between semantic and syntactic features and brain activity through computational methods.

The main contributions of this thesis are summarized as follows.

1. The collection of a neuroimaging dataset under Chinese naturalistic language stimuli

Brain activation during language understanding is fundamental for studying the brain mechanism of language, and the quality of the data heavily constrains what can be learned. Research under naturalistic language stimuli depends on computational models to establish the mapping between language stimuli and brain activation, which has high requirements for the scale and quality of neuroimaging data. Therefore, this thesis collects functional magnetic resonance image data from 12 native Chinese participants when they are listening to 60 Chinese stories with a total duration of about 5 hours. The collected neuroimaging data is then preprocessed and technically validated. The technical validation results show the high quality of the data. In addition, this thesis annotates the linguistic features of the stimulus story text, which lays a foundation for the analysis of the brain representation mechanism of syntactic and semantics.

2. The investigation of the word syntactic representation in the brain by a feature elimination method

Word is the smallest linguistic unit that can be independently used and the foundation that makes up complex phrases and sentences. Studying how words form phrases and sentences must first categorize words and understand the relationships between words. Therefore, this thesis selects three syntactic features: part-of-speech, dependency relationship, and predicate-argument structure, and analyzes the word-level syntactic feature representation patterns of the brain in both Chinese and English. The main difficulty of studying the above problem is to separate a specific syntactic from the others and from semantics. This thesis proposes a feature elimination method to separate different syntactic features from the word embeddings computed by the distributed text representation model. The proposed feature elimination method can eliminate a specific feature from the word embedding space while retaining other features. Based on the original and one-feature-removed word embeddings, we explore how the brain encodes syntactic features by associating these vectors with brain imaging data. The motivation for removing one feature from representations is that if a specific feature is removed from the original word embeddings and if this feature is represented in the brain, the predictability of the brain areas associated with this feature will be severely damaged.  Results suggest some possible contributions of several brain regions to the complex division of syntactic processing, and there are certain overlapping and dissociation differences between Chinese and English.

3. The investigation of the structure-adaptive brain mechanism for the construction of hierarchical structures

The hierarchical nature of language enables finite words to combine into infinite sentences following syntactic rules. In view of this characteristic of language, this thesis studies how the brain builds hierarchical syntactic structure in Chinese and English. In order to explore the general brain mechanism of syntactic structure building, this thesis starts from the structural difference between Chinese and English and quantitatively analyzes the relationship between the branching direction and the working memory load generated by different parsing strategies. The results show that there are obvious differences in the language structure between Chinese and English, with English mainly being right-branching, while Chinese has both left- and right-branching structures. This difference leads to different working memory burdens for different parsing strategies. For Chinese, the bottom-up strategy causes less memory burden, while for English, on the contrary, the top-down strategy has a smaller memory load. Subsequently, through an fMRI analysis, this thesis finds that the brain activation of both Chinese and English participants is more consistent with the parsing strategy with less memory load, indicating that the brain is limited by cognitive resources and adopts parsing strategies with less memory load according to different language structures.

4. The investigation of the brain basis of semantics with a set of mental-experience-based semantic features 

The brain basis of semantics is a common concern of multiple disciplines such as cognitive neuroscience and psychology. Studies have shown that the semantic acquisition of the brain is contributed by mental experience, including concrete sensory-motor experiences and abstract emotional and social experiences. However, most of the existing studies have focused only on one type of experience, ignoring the importance of another type of experience to semantic representation. This thesis selects a representative set of sensory-motor and non-sensory-motor semantic features, including visual, motor, emotion, socialness, space, and time, and manually annotated the scores of each dimension for words in Chinese naturalistic language stimuli. Then, this thesis establishes the mapping between this set of semantic features and brain activation through linear regression models and analyzes the dissociation and overlapping neural patterns of these semantic dimensions. The results show multiple brain regions sensitive to semantic information and reveal a complex semantic atlas in the brain, which provide new insights into the organization pattern of the brain semantic system.