CASIA OpenIR  > 紫东太初大模型研究中心
Bridging the Gap between Different Vocabularies for LLM Ensemble
徐杨一帆1,2; Lu JL(陆金梁)1,2; Zhang JJ(张家俊)1,2,3,4
2024-06
Conference Name2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics
Conference DateJune 16–21, 2024
Conference PlaceMexico City, Mexico
PublisherAssociation for Computational Linguistics
Abstract

Ensembling different large language models (LLMs) to unleash their complementary potential and harness their individual strengths is highly valuable. Nevertheless, vocabulary discrepancies among various LLMs have constrained previous studies to either selecting or blending completely generated outputs. This limitation hinders the dynamic correction and enhancement of outputs during the generation process, resulting in a limited capacity for effective ensemble. To address this issue, we propose a novel method to Ensemble LLMs via Vocabulary Alignment (EVA). EVA bridges the lexical gap among various LLMs, enabling meticulous ensemble at each generation step. Specifically, we first learn mappings between the vocabularies of different LLMs with the assistance of overlapping tokens. Subsequently, these mappings are employed to project output distributions of LLMs into a unified space, facilitating a fine-grained ensemble. Finally, we design a filtering strategy to exclude models that generate unfaithful tokens. Experimental results on commonsense reasoning, arithmetic reasoning, machine translation, and data-to-text generation tasks demonstrate the superiority of our approach compared with individual LLMs and previous ensemble methods conducted on complete outputs. Further analyses confirm that our approach can leverage knowledge from different language models and yield consistent improvement.

Indexed ByEI
Language英语
Sub direction classification自然语言处理
planning direction of the national heavy laboratory语音语言处理
Paper associated data
Document Type会议论文
Identifierhttp://ir.ia.ac.cn/handle/173211/57391
Collection紫东太初大模型研究中心
Corresponding AuthorZhang JJ(张家俊)
Affiliation1.School of Artificial Intelligence, University of Chinese Academy of Sciences
2.Institute of Automation, Chinese Academy of Sciences
3.Wuhan AI Research
4.Shanghai Artificial Intelligence Laboratory
First Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Corresponding Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
徐杨一帆,Lu JL,Zhang JJ. Bridging the Gap between Different Vocabularies for LLM Ensemble[C]:Association for Computational Linguistics,2024.
Files in This Item: Download All
File Name/Size DocType Version Access License
eva.pdf(1982KB)会议论文 开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[徐杨一帆]'s Articles
[Lu JL(陆金梁)]'s Articles
[Zhang JJ(张家俊)]'s Articles
Baidu academic
Similar articles in Baidu academic
[徐杨一帆]'s Articles
[Lu JL(陆金梁)]'s Articles
[Zhang JJ(张家俊)]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[徐杨一帆]'s Articles
[Lu JL(陆金梁)]'s Articles
[Zhang JJ(张家俊)]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: eva.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.