CASIA OpenIR  > 模式识别国家重点实验室  > 自然语言处理
基于篇章建模的机器阅读理解技术研究
田志兴
Subtype博士
Thesis Advisor赵军
2021-05-27
Degree Grantor中国科学院自动化研究所
Place of Conferral中国科学院自动化研究所
Degree Discipline计算机应用技术
Keyword机器阅读理解 篇章建模 句间关系 篇章场景 篇章主题
Abstract

语言理解是认知智能的一个重要体现,同时也是自然语言处理领域一个长期的富有挑战性的目标。为了更加灵活且全面地评价一个系统的语言理解能力,研究者提出了机器阅读理解(Machine Reading Comprehension, MRC)任务。该任务在形式上表现为基于文本的问答,即给定一篇文档,要求机器回答与文档相关的问题。
近年来,得益于深度学习技术的发展以及标注数据规模的提升,机器阅读理解任务得以快速发展。与此同时,模型已经初步掌握了一些基本的文本理解技巧,尤其是在字词的理解以及句子级别的匹配方面,模型已经有了较强的能力。但由于只关注词语或句子建模,现有方法在需要对篇章整体信息建模的场景中仍有很大的局限性。其实验上的表现为:在某些对句间关系建模、事件因果推理等能力有较高要求的测试样例中,模型的表现仍明显弱于人类。因此,本文从篇章建模的角度对机器阅读理解技术展开系统性的研究。整体而言,本文从句间关系、事件关系、篇章主题三种篇章级信息入手,分别在不限定文档形式、限定文档形式为叙事文档以及社交媒体文档的设定下,研究基于篇章建模的机器阅读理解技术。
主要的创新点和研究成果包括:
1、提出了一种基于句间关系建模的机器阅读理解方法
在现有的机器阅读理解方法中,作为篇章级信息之一的句间关系,常常在文档建模时被忽略。这使得模型对文档的整体语义建模不充分,进而影响答案的推理。针对该问题,本文提出了一种基于图编码的多角度建模句间关系的方法。该方法以篇章中的句子为节点,利用图结构刻画句间关系。一方面,从主题关联、语义相似度、篇章内距离三个角度,以静态的方式构建句间关系图;另一方面,为了捕获以上预先设计的角度所不能覆盖的句间关系,该方法还包含了一种动态构图方式。进一步的,通过针对性的图编码和表示融合完成句间关系建模,进而辅助模型对文档的理解。在机器阅读理解子任务——答案句选择任务中进行实验验证。相应的,基于句间关系建模方法,使用强弱不同的底层表示分别构建了两种答案句选择模型。对应实验结果表明该句间关系建模方法有良好的有效性和通用性,且在对抗输入测试中,展现出了良好的鲁棒性。

2、提出了一种基于篇章场景建模的机器阅读理解方法
作为常见文本体裁之一,叙事型文本的理解在机器阅读理解任务中是不可忽视的。该类文本区别于其他形式文本的一个重要特征是其由一系列相互关联的事件组成,因此从事件角度对此类文本进行篇章级建模是必要的,但是现有方法普遍未关注到这一要点。针对此问题,本文受人类阅读行为的启发,提出了一种针对叙事型文本的篇章级事件场景建模方法。该方法引入包括事件因果、人物角色属性在内的事件关联知识,构建由多个事件组成的篇章场景图,并对其进行针对性的编码以完成篇章场景的建模,进而辅助模型对文档的理解。在叙事型机器阅读理解数据集中验证其效果,实验结果证明了该方法的有效性。

3、提出了一种基于篇章主题建模的机器阅读理解方法
社交媒体文本是当今互联网时代重要的文本形式之一。由于作者在发布该类文本时会假设读者与其有类似的背景知识,因此该类文本的篇幅一般较短。这导致社交媒体文本的信息自包含能力较弱,机器阅读理解模型往往难以理解文本所描述的主题,根据该文本回答问题便更加难以做到。因此,在此类场景下模型需首要解决的就是篇章主题建模的问题,但该问题现有方法少有关注。针对该问题,本文提出了一个引入外部知识进行篇章主题建模的机器阅读理解方法。该方法立足于社交媒体文本“主题信息聚集”的特点,以社交媒体平台中其他相关文本为知识源,获取、提炼主题知识,并最终将其融入到篇章的表示中以完成篇章主题建模,进而辅助模型对文档的理解。在相关公开数据集上的实验结果表明,该方法能够通过有效的篇章主题建模来提升文档理解和答案推理能力。

Other Abstract

Language understanding is an important manifestation of cognitive intelligence, and it is also a long-term challenging goal in the field of natural language processing. To evaluate the language comprehension ability of a system more flexibly and comprehensively, the researchers proposed the task of Machine Reading Comprehension (MRC). Formally, the task is defined as a text-based Question Answering, that is, given a document, the machine is required to answer the questions related to the document.

Thanks to the increase in deep learning technology and the scale of annotated data, MRC task has developed rapidly over the past few years. At the same time, the model has initially learned some basic text comprehension skills, especially in terms of word understanding and sentence-level matching. However, as only focusing on the modeling of words or sentences, the existing methods still have great limitations in the scenarios where it is required to model the overall information of the given document. The experimental manifestation is that in certain test 
instances with higher requirements for sentence relations modeling or event causal reasoning, the performance of the model is still significantly weaker than that of humans. Therefore, this article conducts systematic research on Machine Reading Comprehension technology from the perspective of document modeling. On the whole, this article considers three types of document-level information: sentence relations, event relations, and document topic, and studies document modeling based MRC under the setting of unrestricted document form and restricted document form as narrative documents and social media documents, respectively. The main innovations and research results include:

1. A sentence relation modeling based MRC method

In the existing MRC methods, the sentence relations, one of the document-level information, are often ignored during modeling. This results in insufficient modeling of the overall semantics of the document, which in turn affects the reasoning of the answer. To this end, this paper proposes a graph based method which models sentence relations from multiple perspectives. This method takes the sentences in the document as nodes and utilizes graph structure to describe the relationship between sentences. On the one hand, the method builds the relation graphs in a static way, from the perspectives of topic relevance, semantic similarity, and distance within document. On the other hand, in order to capture the relationship between sentences that cannot be covered by the above pre-designed perspective, a dynamic building method is also introduced in the method. This paper conduct experiments on the answer sentence selection task, a subtask of MRC, meanwhile based on the module of sentence relation modeling, two types of answer sentence selection model are built by employing weak and strong underlying representation, respectively. The results demonstrate the effectiveness and versatility of the proposed method that models the relations between sentences. Moreover, the adversarial test shows the robustness of the method.

2. A document scene modeling based MRC method

Narrative is one of the common text genres, and its comprehension should not be ignored in the task of Machine Reading Comprehension. Narrative document is composed of a series of interrelated events, which is its important feature distinguishing from other forms of text. Therefore, it is necessary to model narrative from the perspective of events at document level, but the existing methods generally do not pay attention to this point. In response to this problem, this paper proposes a document-level event scene modeling method for narrative document. The method is inspired by human reading behavior. By introducing 
event association knowledge, such as event causal and person attribute, and utilizing structured description, it assists the MRC model to restore the scene related to the document. The paper conduct experiments on the narrative MRC dataset, and the results prove the effectiveness of the method.
3、A document topic modeling based MRC method

 Social media text is an important text form in nowadays Internet era. The author usually posts a message on the assumption that the readers have specific background knowledge, thus those messages are generally short. This leads to the weak self-containment ability of the document. Moreover, an MRC model suffers from understanding the topics described in the document, so it will be even more difficult to answer questions based on the document. Therefore, in such scenarios, the first problem that the model needs to solve is topic modeling, but the existing methods have seldom paid attention to this problem. To this end, this paper proposes an MRC method that introduces external knowledge to model document topic. The method starts with the characteristics of “topic information clustering” of social media, treats other relevant texts in social media platforms as knowledge sources to acquire and refine topic knowledge, and finally incorporates the topic knowledge of document. Experimental results on relevant public dataset show that this method can improve the ability of document understanding and answer reasoning  through effective document topic modeling.

Pages128
Language中文
Document Type学位论文
Identifierhttp://ir.ia.ac.cn/handle/173211/44817
Collection模式识别国家重点实验室_自然语言处理
Recommended Citation
GB/T 7714
田志兴. 基于篇章建模的机器阅读理解技术研究[D]. 中国科学院自动化研究所. 中国科学院自动化研究所,2021.
Files in This Item:
File Name/Size DocType Version Access License
基于篇章建模的机器阅读理解.pdf(3319KB)学位论文 开放获取CC BY-NC-SA
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[田志兴]'s Articles
Baidu academic
Similar articles in Baidu academic
[田志兴]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[田志兴]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.