CASIA OpenIR  > 模式识别国家重点实验室  > 自然语言处理
Attention With Sparsity Regularization for Neural Machine Translation and Summarization
Zhang, Jiajun; Zhao, Yang; Li, Haoran; Zong, Chengqing
Source PublicationIEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING
2019-03
Volume27Issue:3Pages:507-518
Abstract

The attention mechanism has become the de facto standard component in neural sequence to sequence tasks, such as machine translation and abstractive summarization. It dynamically determines which parts in the input sentence should be focused on when generating each word in the output sequence. Ideally, only few relevant input words should be attended to at each decoding time step and the attention weight distribution should be sparse and sharp. However, previous methods have no good mechanism to control this attention weight distribution. In this paper, we propose a sparse attention model in which a sparsity regularization term is designed to augment the objective function. We explore two kinds of regularizations: L∞-norm regularization and minimum entropy regularization, both of which aim to sharpen the attention weight distribution. Extensive experiments on both neural machine translation and abstractive summarization demonstrate that our proposed sparse attention model can substantially outperform the strong baselines. And the detailed analyses reveal that the final attention distribution indeed becomes sparse and sharp.

KeywordMachine Translation Attention
Language英语
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/23190
Collection模式识别国家重点实验室_自然语言处理
Corresponding AuthorZhang, Jiajun
Affiliation中国科学院自动化研究所
First Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Corresponding Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
Zhang, Jiajun,Zhao, Yang,Li, Haoran,et al. Attention With Sparsity Regularization for Neural Machine Translation and Summarization[J]. IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING,2019,27(3):507-518.
APA Zhang, Jiajun,Zhao, Yang,Li, Haoran,&Zong, Chengqing.(2019).Attention With Sparsity Regularization for Neural Machine Translation and Summarization.IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING,27(3),507-518.
MLA Zhang, Jiajun,et al."Attention With Sparsity Regularization for Neural Machine Translation and Summarization".IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING 27.3(2019):507-518.
Files in This Item: Download All
File Name/Size DocType Version Access License
Attention With Spars(3627KB)期刊论文作者接受稿开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Zhang, Jiajun]'s Articles
[Zhao, Yang]'s Articles
[Li, Haoran]'s Articles
Baidu academic
Similar articles in Baidu academic
[Zhang, Jiajun]'s Articles
[Zhao, Yang]'s Articles
[Li, Haoran]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Zhang, Jiajun]'s Articles
[Zhao, Yang]'s Articles
[Li, Haoran]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: Attention With Sparsity Regularization for Neural Machine Translation and Summarization.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.