CASIA OpenIR  > 模式识别国家重点实验室  > 自然语言处理
Attention With Sparsity Regularization for Neural Machine Translation and Summarization
Zhang, Jiajun; Zhao, Yang; Li, Haoran; Zong, Chengqing
发表期刊IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING
2019-03
卷号27期号:3页码:507-518
摘要

The attention mechanism has become the de facto standard component in neural sequence to sequence tasks, such as machine translation and abstractive summarization. It dynamically determines which parts in the input sentence should be focused on when generating each word in the output sequence. Ideally, only few relevant input words should be attended to at each decoding time step and the attention weight distribution should be sparse and sharp. However, previous methods have no good mechanism to control this attention weight distribution. In this paper, we propose a sparse attention model in which a sparsity regularization term is designed to augment the objective function. We explore two kinds of regularizations: L∞-norm regularization and minimum entropy regularization, both of which aim to sharpen the attention weight distribution. Extensive experiments on both neural machine translation and abstractive summarization demonstrate that our proposed sparse attention model can substantially outperform the strong baselines. And the detailed analyses reveal that the final attention distribution indeed becomes sparse and sharp.

关键词Machine Translation Attention
语种英语
文献类型期刊论文
条目标识符http://ir.ia.ac.cn/handle/173211/23190
专题模式识别国家重点实验室_自然语言处理
通讯作者Zhang, Jiajun
作者单位中国科学院自动化研究所
第一作者单位中国科学院自动化研究所
通讯作者单位中国科学院自动化研究所
推荐引用方式
GB/T 7714
Zhang, Jiajun,Zhao, Yang,Li, Haoran,et al. Attention With Sparsity Regularization for Neural Machine Translation and Summarization[J]. IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING,2019,27(3):507-518.
APA Zhang, Jiajun,Zhao, Yang,Li, Haoran,&Zong, Chengqing.(2019).Attention With Sparsity Regularization for Neural Machine Translation and Summarization.IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING,27(3),507-518.
MLA Zhang, Jiajun,et al."Attention With Sparsity Regularization for Neural Machine Translation and Summarization".IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING 27.3(2019):507-518.
条目包含的文件
条目无相关文件。
个性服务
推荐该条目
保存到收藏夹
查看访问统计
导出为Endnote文件
谷歌学术
谷歌学术中相似的文章
[Zhang, Jiajun]的文章
[Zhao, Yang]的文章
[Li, Haoran]的文章
百度学术
百度学术中相似的文章
[Zhang, Jiajun]的文章
[Zhao, Yang]的文章
[Li, Haoran]的文章
必应学术
必应学术中相似的文章
[Zhang, Jiajun]的文章
[Zhao, Yang]的文章
[Li, Haoran]的文章
相关权益政策
暂无数据
收藏/分享
所有评论 (0)
暂无评论
 

除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。