Knowledge Commons of Institute of Automation,CAS
HiCMAE: Hierarchical Contrastive Masked Autoencoder for self-supervised Audio-Visual Emotion Recognition | |
Licai Sun; Zheng Lian; Bin Liu; Jianhua Tao | |
发表期刊 | Information Fusion |
ISSN | 1566-2535 |
2024 | |
卷号 | 108页码:1-20 |
通讯作者 | Liu, Bin(liubin@nlpr.ia.ac.cn) ; Tao, Jianhua(jhtao@tsinghua.edu.cn) |
摘要 | Audio-Visual Emotion Recognition (AVER) has garnered increasing attention in recent years for its critical role in creating emotionaware intelligent machines. Previous efforts in this area are dominated by the supervised learning paradigm. Despite significant progress, supervised learning is meeting its bottleneck due to the longstanding data scarcity issue in AVER. Motivated by recent advances in self-supervised learning, we propose Hierarchical Contrastive Masked Autoencoder (HiCMAE), a novel self-supervised framework that leverages large-scale self-supervised pre-training on vast unlabeled audio-visual data to promote the advancement of AVER. Following prior arts in self-supervised audio-visual representation learning, HiCMAE adopts two primary forms of selfsupervision for pre-training, namely masked data modeling and contrastive learning. Unlike them which focus exclusively on top-layer representations while neglecting explicit guidance of intermediate layers, HiCMAE develops a three-pronged strategy to foster hierarchical audio-visual feature learning and improve the overall quality of learned representations. Firstly, it incorporates hierarchical skip connections between the encoder and decoder to encourage intermediate layers to learn more meaningful representations and bolster masked audio-visual reconstruction. Secondly, hierarchical cross-modal contrastive learning is also exerted on intermediate representations to narrow the audio-visual modality gap progressively and facilitate subsequent cross-modal fusion. Finally, during downstream fine-tuning, HiCMAE employs hierarchical feature fusion to comprehensively integrate multi-level features from different layers. To verify the effectiveness of HiCMAE, we conduct extensive experiments on 9 datasets covering both categorical and dimensional AVER tasks. Experimental results show that our method significantly outperforms state-of-the-art supervised and self-supervised audio-visual methods, which indicates that HiCMAE is a powerful audio-visual emotion representation learner. Codes and models are publicly available at https://github.com/sunlicai/HiCMAE. |
关键词 | Audio-Visual Emotion Recognition Self-supervised learning Masked autoencoder Contrastive learning |
DOI | 10.1016/j.inffus.2024.102382 |
关键词[WOS] | DEEP ; FEATURES ; AUDIO |
收录类别 | SCI |
语种 | 英语 |
资助项目 | National Natural Science Foundation of China (NSFC)[62201572] ; National Natural Science Foundation of China (NSFC)[62276259] ; National Natural Science Foundation of China (NSFC)[U21B2010] ; National Natural Science Foundation of China (NSFC)[62271083] ; National Natural Science Foundation of China (NSFC)[62306316] ; National Natural Science Foundation of China (NSFC)[62322120] |
项目资助者 | National Natural Science Foundation of China (NSFC) |
WOS研究方向 | Computer Science |
WOS类目 | Computer Science, Artificial Intelligence ; Computer Science, Theory & Methods |
WOS记录号 | WOS:001220967900001 |
出版者 | ELSEVIER |
七大方向——子方向分类 | 智能交互 |
国重实验室规划方向分类 | 人机混合智能 |
是否有论文关联数据集需要存交 | 否 |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/57082 |
专题 | 多模态人工智能系统全国重点实验室 |
作者单位 | 1.School of Artificial Intelligence, University of Chinese Academy of Sciences 2.Institute of Automation, Chinese Academy of Sciences 3.cDepartment of Automation, Tsinghua University 4.Beijing National Research Center for Information Science and Technology, Tsinghua University |
推荐引用方式 GB/T 7714 | Licai Sun,Zheng Lian,Bin Liu,et al. HiCMAE: Hierarchical Contrastive Masked Autoencoder for self-supervised Audio-Visual Emotion Recognition[J]. Information Fusion,2024,108:1-20. |
APA | Licai Sun,Zheng Lian,Bin Liu,&Jianhua Tao.(2024).HiCMAE: Hierarchical Contrastive Masked Autoencoder for self-supervised Audio-Visual Emotion Recognition.Information Fusion,108,1-20. |
MLA | Licai Sun,et al."HiCMAE: Hierarchical Contrastive Masked Autoencoder for self-supervised Audio-Visual Emotion Recognition".Information Fusion 108(2024):1-20. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
1-s2.0-S156625352400(2281KB) | 期刊论文 | 作者接受稿 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论