毕业生知识库

计算机与物联网技术的发展，使得医疗、日常事件相关的数据类型和数据规模迅速增长，促进了以用户个体为中心的健康大数据形成。面向规模庞大、结构复杂的个体实时健康数据的相关研究，能够帮助人们更加科学地分析与管理自身的行为，提升身体健康与生活质量。个体实时健康数据的表示学习已经成为健康大数据研究领域的一个核心研究课题。

本文聚焦面向健康管理的多源异质数据表示学习研究与应用，通过解决多源异质数据流的统一表征、缺失模态数据的重建特征表示和样本稀缺条件下的多模态特征融合三个挑战，形成完善的跨模态高维数据流融合与特征学习的研究模式。本文的主要工作包括：

（1）多源异质数据流的特征学习研究。以用户个体为中心的实时健康数据具有数据来源不同、模态表示差异大的特点，融合实时行为数据时还需要处理不同源数据的采集频率不一致、同源数据内部的采集频率变化问题。本文针对多源异质数据流的统一表征问题，提出了一种时间引导的高阶注意力融合模型，全面、综合地建模多模态数据流中不同模态数据之间、不同时刻数据之间的关联关系。模型主要考虑单模态不规则序列内部的1阶关系，多模态数据之间的2阶相关性以及多模态序列数据之间的3阶关系，然后使用注意力融合模块综合考虑3种数据关系在特征融合中重要程度，最后得到多源异质数据的统一表征。在MIMIC-III和PPMI两个数据集上进行的两个不同健康状况预测任务，验证了模型针对不同数据类型和不同任务都具备有效性。

（2）多源数据的互补特征学习研究。已有的多源数据融合方法主要关注在多种模态数据同时存在的情形下进行数据的匹配和融合，无法解决实际情况中的数据源缺失问题。本文针对多源数据困难存在数据缺失的情况，提出了一种同时包含多个数据源信息的全息特征（Holographic Feature）概念和对应的全息特征学习模型。该模型通过学习构建包含各自数据源信息的记忆模块，然后使用单个数据源的数据从记忆模块中幻想出缺失数据源的信息，从而构建全息特征。实验中进行了基于第一视角和第三视角视频的跨视角行为识别任务，在Charades-Ego和EPIC-MPII数据集上的进行监督和无监督两个实验的实验结果验证了全息特征学习的有效性。

（3）知识驱动的多模态特征学习研究。记录日常行为事件的多模态数据存在标注数据稀缺的问题，难以依靠数据驱动的深度学习模型进行有效的多模态特征融合。本文提出了一个知识驱动的多模态特征学习模型，并应用到了多模态行为识任务中。该模型从多模态数据中提取有效的行为与物体概念，结合外部知识库建立概念之间的图谱关系。最后在外部知识辅助的条件下，使用图神经网络进行概念的动态推理建模和行为预测。在Multimodal、Stanford-ECM和DataEgo三个多模态行为识别数据集上的实验结果验证了模型的有效性。

综上所述，本文通过解决个体实时健康数据中的上述三个挑战，形成了有效的特征学习研究模式，通过充分、有效地从数据和外部知识库中发现、利用有价值的信息，为后续健康管理与决策做出了基础研究支持。

The development of computer and Internet of Things technology has caused the rapid growth of data related to medical and daily events, which promotes the formation of health big data centered on individual users.
Relevant  research on real-time health data of individuals with large scale and complex structure can help people analyze and manage their behavior more scientifically, and improve their health and quality of life.
Representation learning of individual real-time health data has become a core research topic in the field of health big data research.

This article focuses on the research and application of representation learning based on multi-source heterogeneous data for health management.
By solving the three challenges of unified representation learning of multi-source heterogeneous data streams, reconstruction feature representation of missing modal data, and multi-modal feature fusion with insufficient labeled data, this article forms a well-established research pattern of cross-modal high-dimensional data flow fusion and representation learning.
The main work of this article includes:

(1) Research on feature learning of multi-modal heterogeneous data streams.
Individual real-time health data has the characteristics of multiple data sources and large differences in data representation.
It is also necessary to deal with the problems of inconsistent data collection frequency of difference data sources, and the collection frequency inner sequence of one data source.
To solve the above challenges, this article proposes a time-guided high-order attention model, which comprehensively models the association relations between different data modalities in different time steps. The model mainly considers the first-order relationship within the single-mode irregular sequence, the second-order correlation between the multi-modal data and the third-order relationship between the multi-modal sequence data. Then the model uses an attention fusion module to comprehensively compute the important weights of the three types of data relations in feature fusion process. Two different experiments conducted on the MIMIC-III and PPMI datasets verify that the model is effective for different data types in different tasks.

(2) Research on complementary feature learning of multi-source data.
Existing multi-source data fusion methods mainly focus on matching and fusing data in the presence of multiple data sources at the same time, which cannot solve the problem of missing data sources in actual situations.
This article solves the problem where there is a lack of data in multi-source data by proposing a concept named Holographic Feature containing all information of multiple data sources and corresponding model for holographic feature learning.
The model firstly builds two memory modules which store the memory information from different data sources, and then uses the data of single source to hallucinate the missing information from another memory module to construct holographic feature. The hallucinated feature is finally used for activity recognition task.
Two experiments with supervised and unsupervised setting are performed on the Charades-Ego and EPIC-MPII datasets. The experimental results verify the effectiveness of holographic feature learning.

(3) Research on knowledge-driven multimodal feature learning.
Due to the insufficient of labeled multimodal data which records daily events of individuals, it is difficult to rely on data-driven deep learning models for effective multimodal feature fusion.
This article proposes a knowledge-driven multimodal feature learning model and applies it to multimodal activity recognition task. The model firstly extracts activity and object concepts from multimodal data, and establishes a relationship graph among concepts by using an external knowledge base. Finally, with the help of external knowledge, graph neural networks are used to model the dynamic varying patterns of concepts for activity prediction. The experimental results on the three multimodal activity recognition datasets, Multimodal, Stanford-ECM and DataEgo, verify the effectiveness of the model.

In summary, this article has formed a research pattern for effective representation learning by solving the above three challenges in individual real-time health data.
By fully and effectively discovering and using valuable information from data and external knowledge, this article has provided basic research support for subsequent health management and decision-making.