Knowledge Commons of Institute of Automation,CAS
CTNet: Conversational Transformer Network for Emotion Recognition | |
Lian, Zheng1,2![]() ![]() ![]() | |
发表期刊 | IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING
![]() |
2021 | |
期号 | 29页码:985-1000 |
摘要 | Emotion recognition in conversation is a crucial topic for its widespread applications in the field of human-computer interactions. Unlike vanilla emotion recognition of individual utterances, conversational emotion recognition requires modeling both context-sensitive and speaker-sensitive dependencies. Despite the promising results of recent works, they generally do not leverage advanced fusion techniques to generate the multimodal representations of an utterance. In this way, they have limitations in modeling the intra-modal and cross-modal interactions. In order to address these problems, we propose a multimodal learning framework for conversational emotion recognition, called conversational transformer network (CTNet). Specifically, we propose to use the transformer-based structure to model intra-modal and cross-modal interactions among multimodal features. Meanwhile, we utilize word-level lexical features and segment-level acoustic features as the inputs, thus enabling us to capture temporal information in the utterance. Additionally, to model context-sensitive and speaker-sensitive dependencies, we propose to use the multi-head attention based bi-directional GRU component and speaker embeddings. Experimental results on the IEMOCAP and MELD datasets demonstrate the effectiveness of the proposed method. Our method shows an absolute 2.1 similar to 6.2% performance improvement on weighted average F1 over state-of-the-art strategies. |
关键词 | Emotion recognition Context modeling Feature extraction Fuses Speech processing Data models Bidirectional control Context-sensitive modeling conversational transformer network (CTNet) conversational emotion recognition multimodal fusion speaker-sensitive modeling |
DOI | 10.1109/TASLP.2021.3049898 |
收录类别 | SCI |
语种 | 英语 |
资助项目 | National Key Research and Development Plan of China[2018YFB1005003] ; National Natural Science Foundation of China (NSFC)[61831022] ; National Natural Science Foundation of China (NSFC)[61771472] ; National Natural Science Foundation of China (NSFC)[61901473] ; National Natural Science Foundation of China (NSFC)[61773379] ; Inria-CAS Joint Research Project[173211KYSB20170061] ; Inria-CAS Joint Research Project[173211KYSB20190049] |
项目资助者 | National Key Research and Development Plan of China ; National Natural Science Foundation of China (NSFC) ; Inria-CAS Joint Research Project |
WOS研究方向 | Acoustics ; Engineering |
WOS类目 | Acoustics ; Engineering, Electrical & Electronic |
WOS记录号 | WOS:000626505400001 |
出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
七大方向——子方向分类 | 智能交互 |
引用统计 | |
文献类型 | 期刊论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/44124 |
专题 | 多模态人工智能系统全国重点实验室_智能交互 |
通讯作者 | Tao, Jianhua |
作者单位 | 1.Chinese Acad Sci, Natl Lab Pattern Recognit, Inst Automat, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100190, Peoples R China 3.CAS Ctr Excellence Brain Sci & Intelligence Techn, Beijing 100190, Peoples R China |
第一作者单位 | 模式识别国家重点实验室 |
通讯作者单位 | 模式识别国家重点实验室 |
推荐引用方式 GB/T 7714 | Lian, Zheng,Liu, Bin,Tao, Jianhua. CTNet: Conversational Transformer Network for Emotion Recognition[J]. IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING,2021(29):985-1000. |
APA | Lian, Zheng,Liu, Bin,&Tao, Jianhua.(2021).CTNet: Conversational Transformer Network for Emotion Recognition.IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING(29),985-1000. |
MLA | Lian, Zheng,et al."CTNet: Conversational Transformer Network for Emotion Recognition".IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING .29(2021):985-1000. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
TASLP.pdf(2230KB) | 期刊论文 | 作者接受稿 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论