Knowledge Commons of Institute of Automation,CAS
User Response Modeling in Reinforcement Learning for Ads Allocation | |
Zhang, Zhiyuan1,2![]() ![]() ![]() | |
2024-05 | |
会议名称 | The ACM Web Conference 2024 |
会议日期 | May 13 - 17, 2024 |
会议地点 | 新加坡 |
摘要 | User response modeling can enhance the learning of user representations and further improve the reinforcement learning (RL) recommender agent. However, as users’ behaviors are influenced by their long-term preferences and short-term stochastic factors (e.g., weather, mood, or fashion trends), it remains challenging for previous works focusing on recurrent neural network-based user response modeling. Meanwhile, due to the dynamic interests of users, it is often unrealistic to assume the dynamics of users are stationary. Drawing inspiration from opponent modeling, we propose a novel network structure, Deep User Q-Network (DUQN), incorporating a user response probabilistic model into the Q-learning ads allocation strategy to capture the effect of the non-stationary user policy on Q-values. Moreover, we utilize the Recurrent State-Space Model (RSSM) to develop the user response model, which includes deterministic and stochastic components, enabling us to fully consider user long-term preferences and short-term stochastic factors. In particular, we design a RetNet version of RSSM (R-RSSM) to support parallel computation. The R-RSSM model can be further used for multi-step predictions to enable bootstrapping over multiple steps simultaneously. Finally, we conduct extensive experiments on a large-scale offline dataset from the Meituan food delivery platform and a public benchmark. Experimental results show that our method yields superior performance to state-of-the-art (SOTA) baselines. |
关键词 | Ads Allocation Reinforcement Learning User Response Modeling |
DOI | 10.1145/3589335.3648310 |
收录类别 | EI |
语种 | 英语 |
七大方向——子方向分类 | 推荐系统 |
国重实验室规划方向分类 | 先进智能应用与转化 |
是否有论文关联数据集需要存交 | 否 |
引用统计 | |
文献类型 | 会议论文 |
条目标识符 | http://ir.ia.ac.cn/handle/173211/57584 |
专题 | 多模态人工智能系统全国重点实验室 |
通讯作者 | Zhang, Qichao |
作者单位 | 1.中国科学院大学 2.中国科学院自动化研究所 3.Meituan Beijing, China |
第一作者单位 | 中国科学院自动化研究所 |
通讯作者单位 | 中国科学院自动化研究所 |
推荐引用方式 GB/T 7714 | Zhang, Zhiyuan,Zhang, Qichao,Wu, Xiaoxu,et al. User Response Modeling in Reinforcement Learning for Ads Allocation[C],2024. |
条目包含的文件 | ||||||
文件名称/大小 | 文献类型 | 版本类型 | 开放类型 | 使用许可 | ||
3589335.3648310.pdf(2077KB) | 会议论文 | 开放获取 | CC BY-NC-SA | 浏览 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论