CASIA OpenIR  > 模式识别国家重点实验室  > 语音交互
User behavior fusion in dialog management with multi-modal history cues
Yang, Minghao; Tao, Jianhua; Chao, Linlin; Li, Hao; Zhang, Dawei; Che, Hao; Gao, Tingli; Liu, Bin
2015-11-01
发表期刊MULTIMEDIA TOOLS AND APPLICATIONS
卷号74期号:22页码:10025-10051
文章类型Article
摘要It enhances user experience by making the talking avatar be sensitive to user behaviors in human computer interaction (HCI). In this study, we combine user's multi-modal behaviors with behaviors' historical information in dialog management (DM) to improve the avatar's sensitivity not only to user explicit behavior (speech command) but also to user supporting expression (emotion and gesture, etc.). In the dialog management, according to the different contributions of facial expression, gesture and head motion to speech comprehension, we divide the user's multi-modal behaviors into three categories: complementation, conflict and independence. The behavior categories could be first automatically obtained from a short-term and time-dynamic (STTD) fusion model with audio-visual input. Different behavior category leads to different avatar's response in later dialog turns. Usually, the conflict behavior reflects user's ambiguous intention (for example: user says "no" while he (her) is smiling). In this case, the trial-and-error schema is adopted to eliminate the conversation ambiguity. For the later dialog process, we divide all the avatar dialog states into four types: "Ask", "Answer", "Chat" and "Forget". With the detection of complementation and independence behaviors, the user supporting expression as well as his (her) explicit behavior could be estimated as triggers for topic maintenance or transfer among four dialog states. At the first section of experiments, we discuss the reliability of STTD model for user behavior classification. Based on the proposed dialog management and STTD model, we continue to construct a drive route information query system by connecting the user behavior sensitive dialog management (BSDM) to a 3D talking avatar. The practical conversation records of avatar with different users show that the BSDM makes the avatar be able to understand and be sensitive to the users' facial expressions, emotional voice and gesture, which improves user experience on multi-modal human computer conversation.
关键词Dialog Management (Dm) Multi-modal Data Fusion Human Computer Interaction (Hci) Emotion Detection
WOS标题词Science & Technology ; Technology
DOI10.1007/s11042-014-2161-5
收录类别SCI
语种英语
项目资助者National Natural Science Foundation of China (NSFC)(61273288 ; Major Program for the National Social Science Fund of China(13ZD189) ; 61233009 ; 61203258 ; 61530503 ; 61332017 ; 61375027)
WOS研究方向Computer Science ; Engineering
WOS类目Computer Science, Information Systems ; Computer Science, Software Engineering ; Computer Science, Theory & Methods ; Engineering, Electrical & Electronic
WOS记录号WOS:000364019400011
引用统计
被引频次:1[WOS]   [WOS记录]     [WOS相关记录]
文献类型期刊论文
条目标识符http://ir.ia.ac.cn/handle/173211/10496
专题模式识别国家重点实验室_语音交互
作者单位Chinese Acad Sci, Natl Lab Pattern Recognit, Inst Automat, Beijing, Peoples R China
推荐引用方式
GB/T 7714
Yang, Minghao,Tao, Jianhua,Chao, Linlin,et al. User behavior fusion in dialog management with multi-modal history cues[J]. MULTIMEDIA TOOLS AND APPLICATIONS,2015,74(22):10025-10051.
APA Yang, Minghao.,Tao, Jianhua.,Chao, Linlin.,Li, Hao.,Zhang, Dawei.,...&Liu, Bin.(2015).User behavior fusion in dialog management with multi-modal history cues.MULTIMEDIA TOOLS AND APPLICATIONS,74(22),10025-10051.
MLA Yang, Minghao,et al."User behavior fusion in dialog management with multi-modal history cues".MULTIMEDIA TOOLS AND APPLICATIONS 74.22(2015):10025-10051.
条目包含的文件 下载所有文件
文件名称/大小 文献类型 版本类型 开放类型 使用许可
2015_MHCI_Multimedia(2008KB)期刊论文作者接受稿开放获取CC BY-NC-SA浏览 下载
个性服务
推荐该条目
保存到收藏夹
查看访问统计
导出为Endnote文件
谷歌学术
谷歌学术中相似的文章
[Yang, Minghao]的文章
[Tao, Jianhua]的文章
[Chao, Linlin]的文章
百度学术
百度学术中相似的文章
[Yang, Minghao]的文章
[Tao, Jianhua]的文章
[Chao, Linlin]的文章
必应学术
必应学术中相似的文章
[Yang, Minghao]的文章
[Tao, Jianhua]的文章
[Chao, Linlin]的文章
相关权益政策
暂无数据
收藏/分享
文件名: 2015_MHCI_Multimedia Tools and Applications_SCI.pdf
格式: Adobe PDF
所有评论 (0)
暂无评论
 

除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。