CASIA OpenIR  > 学术期刊  > Machine Intelligence Research
Offline Pre-trained Multi-agent Decision Transformer
Linghui Meng1,2; Muning Wen3; Chenyang Le3; Xiyun Li1,4; Dengpeng Xing1,2; Weinan Zhang3; Ying Wen3; Haifeng Zhang1,2; Jun Wang6; Yaodong Yang5; Bo Xu1,2
Source PublicationMachine Intelligence Research
ISSN2731-538X
2023
Volume20Issue:2Pages:233-248
AbstractOffline reinforcement learning leverages previously collected offline datasets to learn optimal policies with no necessity to access the real environment. Such a paradigm is also desirable for multi-agent reinforcement learning (MARL) tasks, given the combinatorially increased interactions among agents and with the environment. However, in MARL, the paradigm of offline pre-training with online fine-tuning has not been studied, nor even datasets or benchmarks for offline MARL research are available. In this paper, we facilitate the research by providing large-scale datasets and using them to examine the usage of the decision transformer in the context of MARL. We investigate the generalization of MARL offline pre-training in the following three aspects: 1) between single agents and multiple agents, 2) from offline pretraining to online fine tuning, and 3) to that of multiple downstream tasks with few-shot and zero-shot capabilities. We start by introducing the first offline MARL dataset with diverse quality levels based on the StarCraftII environment, and then propose the novel architecture of multi-agent decision transformer (MADT) for effective offline learning. MADT leverages the transformer′s modelling ability for sequence modelling and integrates it seamlessly with both offline and online MARL tasks. A significant benefit of MADT is that it learns generalizable policies that can transfer between different types of agents under different task scenarios. On the StarCraft II offline dataset, MADT outperforms the state-of-the-art offline reinforcement learning (RL) baselines, including BCQ and CQL. When applied to online tasks, the pre-trained MADT significantly improves sample efficiency and enjoys strong performance in both few-short and zero-shot cases. To the best of our knowledge, this is the first work that studies and demonstrates the effectiveness of offline pre-trained models in terms of sample efficiency and generalizability enhancements for MARL.
KeywordPre-training model multi-agent reinforcement learning (MARL) decision making transformer offline reinforcement learning
DOI10.1007/s11633-022-1383-7
Citation statistics
Cited Times:3[WOS]   [WOS Record]     [Related Records in WOS]
Document Type期刊论文
Identifierhttp://ir.ia.ac.cn/handle/173211/55977
Collection学术期刊_Machine Intelligence Research
Affiliation1.Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
2.School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
3.Shanghai Jiao Tong University, Shanghai 200240, China
4.School of Future Technology, University of Chinese Academy of Sciences, Beijing 100049, China
5.Institute for AI, Peking University, Beijing 100871, China
6.Department of Computer Science, University College London, London WC1E 6BT, UK
First Author AffilicationInstitute of Automation, Chinese Academy of Sciences
Recommended Citation
GB/T 7714
Linghui Meng,Muning Wen,Chenyang Le,et al. Offline Pre-trained Multi-agent Decision Transformer[J]. Machine Intelligence Research,2023,20(2):233-248.
APA Linghui Meng.,Muning Wen.,Chenyang Le.,Xiyun Li.,Dengpeng Xing.,...&Bo Xu.(2023).Offline Pre-trained Multi-agent Decision Transformer.Machine Intelligence Research,20(2),233-248.
MLA Linghui Meng,et al."Offline Pre-trained Multi-agent Decision Transformer".Machine Intelligence Research 20.2(2023):233-248.
Files in This Item: Download All
File Name/Size DocType Version Access License
MIR-2022-07-219.pdf(2121KB)期刊论文出版稿开放获取CC BY-NC-SAView Download
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Linghui Meng]'s Articles
[Muning Wen]'s Articles
[Chenyang Le]'s Articles
Baidu academic
Similar articles in Baidu academic
[Linghui Meng]'s Articles
[Muning Wen]'s Articles
[Chenyang Le]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Linghui Meng]'s Articles
[Muning Wen]'s Articles
[Chenyang Le]'s Articles
Terms of Use
No data!
Social Bookmark/Share
File name: MIR-2022-07-219.pdf
Format: Adobe PDF
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.