[论文解读] Cooperation and Reputation Dynamics with Reinforcement Learning

Nicolas Anastassacos, Julián García+2 more|arXiv (Cornell University)|Feb 15, 2021

Evolutionary Game Theory and Cooperation参考文献 34被引用 6

一句话总结

本文研究了强化学习（RL）智能体如何自主学习建立并协调有效的声誉系统以维持合作。它提出了两种机制——通过引入固定智能体来引导声誉标签，以及基于自我表现的内在内省奖励——以稳定合作，在标准Q-learning收敛至次优均衡的情况下，仍可在完全去中心化的环境中实现高达80%的合作率。

ABSTRACT

Creating incentives for cooperation is a challenge in natural and artificial systems. One potential answer is reputation, whereby agents trade the immediate cost of cooperation for the future benefits of having a good reputation. Game theoretical models have shown that specific social norms can make cooperation stable, but how agents can independently learn to establish effective reputation mechanisms on their own is less understood. We use a simple model of reinforcement learning to show that reputation mechanisms generate two coordination problems: agents need to learn how to coordinate on the meaning of existing reputations and collectively agree on a social norm to assign reputations to others based on their behavior. These coordination problems exhibit multiple equilibria, some of which effectively establish cooperation. When we train agents with a standard Q-learning algorithm in an environment with the presence of reputation mechanisms, convergence to undesirable equilibria is widespread. We propose two mechanisms to alleviate this: (i) seeding a proportion of the system with fixed agents that steer others towards good equilibria; and (ii), intrinsic rewards based on the idea of introspection, i.e., augmenting agents' rewards by an amount proportionate to the performance of their own strategy against themselves. A combination of these simple mechanisms is successful in stabilizing cooperation, even in a fully decentralized version of the problem where agents learn to use and assign reputations simultaneously. We show how our results relate to the literature in Evolutionary Game Theory, and discuss implications for artificial, human and hybrid systems, where reputations can be used as a way to establish trust and cooperation.

研究动机与目标

• 研究RL智能体如何独立学习建立并协调有效的声誉机制以维持合作。
• 识别标准Q-learning在基于声誉的合作博弈中为何无法收敛至高效均衡。
• 评估引入固定智能体和内省内在奖励在引导RL智能体趋向稳定合作均衡方面的有效性。
• 将演化博弈论（EGT）的预测与RL驱动的学习在声誉动态中的预测进行比较。
• 证明结合引入固定智能体与内省机制，可在完全去中心化、自组织的系统中实现高水平的合作。

提出的方法

• 智能体在具有二元声誉（合作者或背叛者）的重复社会困境中使用标准Q-learning进行训练。
• 引入一种声誉系统，其中智能体根据社会规范分配声誉，且行动基于他人的声誉。
• 引入固定比例的具有预设互惠策略的智能体，以引导系统朝向期望的均衡。
• 添加基于智能体与自身副本对战表现的内在奖励，以促进自我一致的合作。
• 在完全去中心化的环境中测试系统，其中智能体同时学习声誉分配和行动策略。
• 分析多种均衡，重点关注哪些规范与行动规则组合能导致稳定合作。

实验结果

研究问题

• RQ1 标准Q-learning算法能否在去中心化的基于声誉的合作博弈中收敛至合作均衡？
• RQ2 尽管存在有效的社会规范，为何RL智能体会收敛至次优均衡？
• RQ3 引入固定智能体在引导RL智能体协调至单一合作声誉规范方面有多有效？
• RQ4 基于内省的内在奖励是否能在缺乏外部协调的情况下稳定合作？
• RQ5 引入固定智能体与内省奖励的组合对在去中心化声誉系统中实现高水平合作有何综合影响？

主要发现

• 即使存在有效的社会规范，标准Q-learning仍频繁收敛至不良均衡，原因在于声誉标签的协调失败。
• 引入50%具有固定互惠策略的智能体显著改善了协调，引导系统朝向支持稳定合作的规范3与行动规则5。
• 基于自我表现的内省奖励促使智能体更倾向于选择合作策略，尤其在与引入固定智能体结合时效果更显著。
• 引入固定智能体与内省奖励的结合在完全去中心化场景中实现了高达80%的合作率，表明两种机制之间存在显著协同效应。
• 研究结果表明，当存在声誉系统时，EGT模型会高估合作潜力，因其假设社会学习而非个体基于RL的探索。
• 稳定性分析显示存在多个均衡，且RL智能体需要外部引导机制以避免在无效声誉标签上发生协调失败。