Learning to Play Football From Sports Domain Perspective: A Knowledge-Embedded Deep Reinforcement Learning Framework

doi:10.1109/TG.2022.3207068

	Learning to Play Football From Sports Domain Perspective: A Knowledge-Embedded Deep Reinforcement Learning Framework
	Liu, Boyin 1,2; Pu, Zhiqiang1,2 ; Zhang, Tianle1,2 ; Wang, Huimu3 ; Yi, Jianqiang1,2 ; Mi, Jiachen 4
发表期刊	IEEE TRANSACTIONS ON GAMES
ISSN	2475-1502
	2023-12-01
卷号	15 期号:4 页码:648-657
通讯作者	Pu, Zhiqiang(zhiqiang.pu@ia.ac.cn)
摘要	Applying deep reinforcement learning to football games has recently received extensive attention. However, this remains challenging due to the excessively high complexity of the football environment, such as high-dynamical game states, sparse rewards, and multiple roles with different capabilities. Existing works aim to address these problems without considering abundant domain knowledge of football. In this article, a football knowledge-embedded learning framework is proposed. Specifically, the pitch control concept is innovatively introduced to design a knowledge-embedded state representation. As a result, a novel pitch control model is designed that quantitatively provides space influence values of a single player, the whole team, and the ball. Different from existing models, this model additionally considers each player's various capabilities, including flexibility, explosive force, and stamina. Furthermore, the deformable convolution network is adopted for state representation extracting, which is used to process the geometric transformation of the players' positions and spatial influence values generated by the pitch control model. Then, based on this comprehensive state representation, a proximal policy optimization-based reinforcement learning scheme is adopted to generate the final policy. Finally, extensive simulations, including learning against a fixed opponent and learning from self-play, clearly show the effectiveness and adaptability of our proposed framework.
关键词	Deformable convolution football analysis pitch control reinforcement learning
DOI	10.1109/TG.2022.3207068
关键词[WOS]	GAME
收录类别	SCI
语种	英语
资助项目	National Key Research, Development Program of China
项目资助者	National Key Research, Development Program of China
WOS研究方向	Computer Science
WOS类目	Computer Science, Artificial Intelligence ; Computer Science, Software Engineering
WOS记录号	WOS:001128375200008
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
引用统计	被引频次：1[WOS] [WOS记录] [WOS相关记录]
文献类型	期刊论文
条目标识符	http://ir.ia.ac.cn/handle/173211/54890
专题	复杂系统认知与决策实验室
通讯作者	Pu, Zhiqiang
作者单位	1.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China 2.Chinese Acad Sci, Inst Automat, Beijing 100190, Peoples R China 3.JD COM, Beijing 100176, Peoples R China 4.Beijing Technol & Business Univ, Sch Artificial Intelligence, Beijing 102401, Peoples R China
第一作者单位	中国科学院自动化研究所
通讯作者单位	中国科学院自动化研究所
推荐引用方式 GB/T 7714	Liu, Boyin,Pu, Zhiqiang,Zhang, Tianle,et al. Learning to Play Football From Sports Domain Perspective: A Knowledge-Embedded Deep Reinforcement Learning Framework[J]. IEEE TRANSACTIONS ON GAMES,2023,15(4):648-657.
APA	Liu, Boyin,Pu, Zhiqiang,Zhang, Tianle,Wang, Huimu,Yi, Jianqiang,&Mi, Jiachen.(2023).Learning to Play Football From Sports Domain Perspective: A Knowledge-Embedded Deep Reinforcement Learning Framework.IEEE TRANSACTIONS ON GAMES,15(4),648-657.
MLA	Liu, Boyin,et al."Learning to Play Football From Sports Domain Perspective: A Knowledge-Embedded Deep Reinforcement Learning Framework".IEEE TRANSACTIONS ON GAMES 15.4(2023):648-657.