Knowledge Commons of Institute of Automation,CAS
Bridging the Gap between Different Vocabularies for LLM Ensemble | |
徐杨一帆1,2![]() ![]() ![]() | |
2024-06 | |
会议名称 | 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics |
会议日期 | June 16–21, 2024 |
会议地点 | Mexico City, Mexico |
出版者 | Association for Computational Linguistics |
摘要 | Ensembling different large language models (LLMs) to unleash their complementary potential and harness their individual strengths is highly valuable. Nevertheless, vocabulary discrepancies among various LLMs have constrained previous studies to either selecting or blending completely generated outputs. This limitation hinders the dynamic correction and enhancement of outputs during the generation process, resulting in a limited capacity for effective ensemble. To address this issue, we propose a novel method to Ensemble LLMs via Vocabulary Alignment (EVA). EVA bridges the lexical gap among various LLMs, enabling meticulous ensemble at each generation step. Specifically, we first learn mappings between the vocabularies of different LLMs with the assistance of overlapping tokens. Subsequently, these mappings are employed to project output distributions of LLMs into a unified space, facilitating a fine-grained ensemble. Finally, we design a filtering strategy to exclude models that generate unfaithful tokens. Experimental results on commonsense reasoning, arithmetic reasoning, machine translation, and data-to-text generation tasks demonstrate the superiority of our approach compared with individual LLMs and previous ensemble methods conducted on complete outputs. Further analyses confirm that our approach can leverage knowledge from different language models and yield consistent improvement. |
收录类别 | EI |
语种 | 英语 |
七大方向——子方向分类 | 自然语言处理 |
国重实验室规划方向分类 | 语音语言处理 |
是否有论文关联数据集需要存交 | 否 |
文献类型 | 会议论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/57391 |
专题 | 紫东太初大模型研究中心 |
通讯作者 | Zhang JJ(张家俊) |
作者单位 | 1.School of Artificial Intelligence, University of Chinese Academy of Sciences 2.Institute of Automation, Chinese Academy of Sciences 3.Wuhan AI Research 4.Shanghai Artificial Intelligence Laboratory |
第一作者单位 | 中国科学院自动化研究所 |
通讯作者单位 | 中国科学院自动化研究所 |
推荐引用方式 GB/T 7714 | 徐杨一帆,Lu JL,Zhang JJ. Bridging the Gap between Different Vocabularies for LLM Ensemble[C]:Association for Computational Linguistics,2024. |
条目包含的文件 | 下载所有文件 | |||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
eva.pdf(1982KB) | 会议论文 | 开放获取 | CC BY-NC-SA | 浏览 下载 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论