CASIA OpenIR  > 毕业生  > 博士学位论文
智能对话系统中的知识表示、激活与利用研究
陈修意
2022-05-18
Pages132
Subtype博士
Abstract

人们在对话交流过程中通过话语的字面意义来传递其真实意图,话语的字面意义是意图的载体,但意图并不完全是话语字面的意义。因此,在人机交互时,智能对话系统不仅需要词法、句法等基本的语言知识来解读话语的字面意义,还需要背景知识、百科知识、领域知识和常识等知识来理解用户的真实意图。此外,作为新一代人机交互界面,智能对话系统需要具备与外部知识库交互的能力,来满足用户的信息获取需求,为用户提供更好的信息服务。自然语言处理中常见的知识形式有结构化知识库和非结构化知识库以及预训练的模型参数,因此,本文以结构化、非结构化和预训练模型参数等典型形式的知识为研究对象,研究对话系统中的知识建模问题,并重点关注其中的三个子问题:面向对话的知识表示问题、对话过程中的知识激活问题以及对话回复时的知识利用问题。


本文的主要研究内容和创新点如下:

1. 基于工作记忆模型的任务型对话生成研究

任务型对话系统依赖于背景知识库信息来帮助用户完成特定任务,现有端到端模型通常采用记忆网络来建模知识库信息。但是这些记忆增强的模型通常混淆对话文本和知识库信息,将其同等对待并存储在一个记忆中。受到心理学工作记忆研究的启发,本文提出一个用于对话回复生成的工作记忆模型。首先,本文采用基于记忆网络的语义记忆与情景记忆来分别编码与存储知识库实体与对话上下文;其次,本文提出的工作记忆模块与两个记忆模块进行迭代式交互,从中激活词汇分布、对话上下文的拷贝分布以及知识库实体的拷贝分布;最后,本文提出基于哨兵思想的词汇选择策略,该策略从三个分布进行词汇选择,将知识库实体恰当地拷贝到对话回复中。此外,本文还提出上下文感知的记忆编码来缓解对话中的未登录词问题。实验结果表明,本文所提方法能够较为准确地激活记忆中的情景词与语义实体,并在对话生成过程中利用这些激活的实体,提升了任务型对话系统的准确性与信息量。

2. 基于后验信息增强的对话知识管理研究

知识对话的前提是知识获取与激活,为此本文研究知识对话系统中的知识选择与对话生成联合建模问题。这是两个相辅相成的子任务:对于相同的对话上下文,不同的人可能会选择不同的知识文本来产生不同的对话回复,以及如果人们预先知道对话回复就能反过来推断知识选择策略。现有主流方法采用基于隐变量模型的知识管理,本文首先分析隐变量模型在先后验知识选择利用上的差异,然后提出基于后验信息增强的对话知识管理器。本方法首先采用键-值记忆网络将文本知识编码为文本级和单词级表示;然后采用后验信息预测模块来增强文本级的知识选择;最后使用基于知识蒸馏的训练策略来消除知识选择的暴露偏差。实验结果表明,该方法能够弥合隐变量模型在先后验知识选择利用上的差异,提升了知识选择的准确率与对话生成的信息量。

3. 基于预训练语言模型的知识对话系统研究

在对话交流过程中,人们会依据外部世界感知的信息以及从记忆中抽取的信息来理解话语,其中记忆分为外显记忆和内隐记忆。近期,预训练语言模型以自监督训练的方式学习大规模文本里的知识,大幅提升了下游任务的性能,分析性实验表明预训练语言模型能够将知识隐式地存储在模型参数中。因此,本文首先在对话生成任务上分析了多领域预训练微调机制的有效性,然后在生成式与检索式两类对话系统中探索如何将预训练模型的隐性知识与显式的结构化知识融合。在生成式对话系统中,本文提出两种知识图谱线性化方式将外部知识融入生成式预训练模型中;在检索式对话系统中,本文使用基于预训练模型的单流编码器将对话文本和知识图谱进行深层交互以选择候选回复。实验结果表明,该方法能够有效地利用显性和隐性知识,提升多领域知识驱动的对话系统的性能。
 

Other Abstract

During the conversation, people pass their true intention via the literal meaning of utterances. Utterances are the carrier of human intention but the intention is not completely the literal meaning of utterances. Therefore, in human-computer interaction, intelligent dialogue systems not only need the basic linguistic knowledge such as morphology and syntax to interpret the literal meaning of utterances, but also require the background knowledge, encyclopedism, domain knowledge and commonsense to understand speaker’s intention. In addition, as a new generation of human-computer interface, intelligent dialogue systems need the ability to interact with the external knowledge base to meet the user’s information acquisition needs and provide better information service for users. There are structured knowledge bases, unstructured text and pretrained model parameters, which is commonly used as knowledge in natural language processing. To this end, with a focus on the typical knowledge representation, such as structured knowledge, unstructured text and network parameters, this paper studies knowledge modeling in intelligent dialogue systems, and mainly focuses on three subproblems: (1) How to obtain the good knowledge representation for dialogue systems, (2) Which knowledge should be activated during conversation and (3) How to incorporate the activated knowledge into dialogue responses.


The main research contents and innovations of this thesis are shown as follows:

1. Research on Task-Oriented Dialogue Generation Based on Working Memory Model

Task-oriented dialogue systems rely on background knowledge base to help users complete specific tasks, and existing end-to-end models usually use memory networks to store the knowledge base. But these memory-augmented models often confuse dialogue text and knowledge information, treating them equally and simply storing them in one memory. Inspired by the psychological studies on working memory, this paper proposes a working memory model for dialog response generation. First, memory network based semantic memory and episodic memory are used to encode and store knowledge entity and dialog context respectively; Secondly, the working memory is proposed to interact with the two memory module and then activate the vocabulary distribution, copy distribution for dialog context and copy distribution for knowledge base entities from these memories; Finally, a rule-based word selection strategy based on the sentinel idea is proposed to choose the token from these three distributions and properly copy the knowledge base entity during the dialogue generation process. Furthermore, context-aware memory encoding is proposed to alleviate the problem of unknown words in dialogue context. Experimental results demonstrate that the proposed method can accurately activate the situational words and semantic entities from the memories, and properly integrate the activated entities into dialogue generation, which improves the accuracy and informativeness of the task-oriented dialogue systems. 

2. Research on Dialogue Knowledge Management Based on Posterior Information Enhancement

The precursor of knowledge-aware dialogue generation is the acquisition and activation of knowledge. For this reason, this paper studies the joint modeling problem of knowledge selection and dialogue generation in knowledge-aware dialogue systems. They are two complementary sub-tasks because there may be diverse knowledge text to generate different responses for the same context which help their selection decisions in turn. The mainstream methods use latent variable models to build the knowledge manager, and this paper first analyzes the gap between prior and posterior knowledge selection for knowledge-grounded dialogue generation and then proposes the posterior information enhanced dialogue knowledge manager to bridge the gap. First, key-value memory network is used to encode the knowledge text as sentence-level and word-level representation; Then, posterior information prediction module is proposed to enhance the sentence-level knowledge selection; Finally, knowledge distillation based training strategy is proposed to remove the exposure bias of knowledge selection. Experimental results show that the proposed method could bridge the gap between prior and posterior knowledge selection for knowledge-grounded dialogue generation, and improves the accuracy of knowledge selection and informativeness of generated responses.

3. Research on Knowledge-Aware Dialogue Systems Based on Pre-Trained Language Models

During the conversation, people will receive external information and extract the relevant information from memory to comprehend the utterance, and memory is divided into explicit memory and implicit memory. Recently, the pre-trained language models learn the knowledge of large-scale text via self-supervision training, which greatly improves the performance of the downstream tasks, and analytical experiments indicate that the pre-trained language models can store the implicit knowledge in the model parameters. Therefore, this paper first analyzes the effectiveness of multi-domain pretraining and fine-tuning mechanism on dialogue generation and then explores how to fuse the explicit structured knowledge and the pre-trained models in the generative and retrieval dialogue systems. In the generative dialogue systems, two kinds of knowledge graph linearization methods are proposed to incorporate the structured knowledge into the generative pre-trained models; In the retrieval dialogue systems, a pre-trained model based single-stream encoder is used to make deep interactions among the dialogue context, response candidate, and external knowledge graph for response selection. Experimental results show that the proposed methods could efficiently incorporate both explicit and implicit knowledge to improve the performance on a multi-domain knowledge-driven dialogue dataset. 

Keyword对话系统,知识表示,知识对话生成
Subject Area自然语言处理
Language中文
Document Type学位论文
Identifierhttp://ir.ia.ac.cn/handle/173211/48666
Collection毕业生_博士学位论文
Recommended Citation
GB/T 7714
陈修意. 智能对话系统中的知识表示、激活与利用研究[D]. 中国科学院自动化研究所. casia,2022.
Files in This Item:
File Name/Size DocType Version Access License
xiuyichen_ucasthesis(5912KB)学位论文 限制开放CC BY-NC-SA
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[陈修意]'s Articles
Baidu academic
Similar articles in Baidu academic
[陈修意]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[陈修意]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.