[论文解读] Emergent Bartering Behaviour in Multi-Agent Reinforcement Learning

Michael Bradley Johanson, Edward Hughes+2 more|arXiv (Cornell University)|May 13, 2022

Complex Systems and Time Series Analysis被引用 8

一句话总结

本文提出了Fruit Market环境，这是一个多智能体强化学习设置，其中智能体通过生产、贸易和消费响应供需关系，自主发展出物物交换行为。关键结果表明，出现了局部定价、通过区域间运输进行套利以及自适应的生产/消费等涌现的微观经济现象，证明了复杂经济行为可自然地从强化学习中涌现，而无需领域特定的编码。

ABSTRACT

Advances in artificial intelligence often stem from the development of new environments that abstract real-world situations into a form where research can be done conveniently. This paper contributes such an environment based on ideas inspired by elementary Microeconomics. Agents learn to produce resources in a spatially complex world, trade them with one another, and consume those that they prefer. We show that the emergent production, consumption, and pricing behaviors respond to environmental conditions in the directions predicted by supply and demand shifts in Microeconomics. We also demonstrate settings where the agents' emergent prices for goods vary over space, reflecting the local abundance of goods. After the price disparities emerge, some agents then discover a niche of transporting goods between regions with different prevailing prices -- a profitable strategy because they can buy goods where they are cheap and sell them where they are expensive. Finally, in a series of ablation experiments, we investigate how choices in the environmental rewards, bartering actions, agent architecture, and ability to consume tradable goods can either aid or inhibit the emergence of this economic behavior. This work is part of the environment development branch of a research program that aims to build human-like artificial general intelligence through multi-agent interactions in simulated societies. By exploring which environment features are needed for the basic phenomena of elementary microeconomics to emerge automatically from learning, we arrive at an environment that differs from those studied in prior multi-agent reinforcement learning work along several dimensions. For example, the model incorporates heterogeneous tastes and physical abilities, and agents negotiate with one another as a grounded form of communication.

研究动机与目标

• 开发一个支持多智能体强化学习的环境，以实现贸易、定价和专业化等涌现的微观经济行为。
• 探究深度强化学习智能体是否能从随机初始化中自主发现复杂的经济行为，如套利和供需调节。
• 识别能够促进或抑制多智能体系统中社会与经济行为涌现的环境设计选择。
• 通过展示最先进的强化学习智能体学习基础经济现象，弥合多智能体强化学习与基于代理的计算经济学之间的差距。

提出的方法

• 智能体在空间结构复杂的环境中运行，根据个体偏好和生产能力，自主进行资源（如苹果、香蕉）的生产、消费和贸易。
• 智能体通过移动、出价和交换动作进行协商，价格基于供需动态自然涌现。
• 环境引入了异质性智能体特征，包括不同的生产能力与消费偏好，以模拟现实世界中的经济多样性。
• 采用V-MPO这一深度强化学习算法进行强化学习，稀疏奖励与消费及饥饿感降低相关。
• 智能体通过分布式训练学习，未对贸易行为进行显式奖励塑造，而是依赖于消费和机会成本带来的涌现激励。
• 该环境作为开源项目发布于Melting Pot套件中，以支持可复现性，并推动多智能体强化学习与计算经济学的进一步研究。

实验结果

研究问题

• RQ1 多智能体强化学习智能体是否能在无显式贸易奖励塑造的情况下自发发展出物物交换行为？
• RQ2 环境中的供需变化如何影响涌现的定价以及生产/消费行为？
• RQ3 资源丰度的空间差异是否导致局部定价并催生套利策略的出现？
• RQ4 哪些环境与架构设计选择是实现复杂经济行为（如贸易与专业化）所必需的？
• RQ5 智能体能否学会在价格差异的区域间运输商品，从而发现套利作为盈利策略？

主要发现

• 智能体自发形成了反映区域资源丰度的局部定价，价格差异在供需变化下自然产生。
• 当区域间出现价格差异时，部分智能体发现了并专精于从低价区域向高价区域运输商品，通过套利获得更高奖励。
• 贸易的涌现对环境设计敏感：移除饥饿惩罚或改变贸易机制（如使用‘丢弃’或‘给予’动作）会显著阻碍或完全阻止贸易的出现。
• 仅使用‘丢弃’和‘给予’动作的智能体未能学会贸易，表明联合探索贸易需要更结构化或可靠的通信机制。
• 消融实验确认，智能体架构、移动惩罚以及奖励塑造（如饥饿惩罚）对经济行为是否涌现具有决定性影响。
• Fruit Market环境成功使最先进的深度强化学习智能体在无领域特定代码或先验知识的情况下，学习到复杂且类人的经济行为，包括生产、消费、贸易与套利。