[论文解读] Learning to Influence Human Behavior with Offline Reinforcement Learning

Joey Hong, Sergey Levine+1 more|arXiv (Cornell University)|Mar 3, 2023

Reinforcement Learning in Robotics被引用 6

一句话总结

本文提出使用离线强化学习（RL）训练AI智能体，在协作任务中影响次优的人类行为，即使没有明确示范影响策略。通过从多样化的人类-人类交互数据中学习并建模人类潜在策略，该方法使智能体能够推断并应用新颖的影响策略——例如通过阻止访问偏好动作来引导行为改变——从而在人类行为策略随时间演变的情况下，仍能显著提升人类表现。

ABSTRACT

When interacting with people, AI agents do not just influence the state of the world -- they also influence the actions people take in response to the agent, and even their underlying intentions and strategies. Accounting for and leveraging this influence has mostly been studied in settings where it is sufficient to assume that human behavior is near-optimal: competitive games, or general-sum settings like autonomous driving alongside human drivers. Instead, we focus on influence in settings where there is a need to capture human suboptimality. For instance, imagine a collaborative task in which, due either to cognitive biases or lack of information, people do not perform very well -- how could an agent influence them towards more optimal behavior? Assuming near-optimal human behavior will not work here, and so the agent needs to learn from real human data. But experimenting online with humans is potentially unsafe, and creating a high-fidelity simulator of the environment is often impractical. Hence, we focus on learning from an offline dataset of human-human interactions. Our observation is that offline reinforcement learning (RL) can learn to effectively influence suboptimal humans by extending and combining elements of observed human-human behavior. We demonstrate that offline RL can solve two challenges with effective influence. First, we show that by learning from a dataset of suboptimal human-human interaction on a variety of tasks -- none of which contains examples of successful influence -- an agent can learn influence strategies to steer humans towards better performance even on new tasks. Second, we show that by also modeling and conditioning on human behavior, offline RL can learn to affect not just the human's actions but also their underlying strategy, and adapt to changes in their strategy.

研究动机与目标

• 解决因认知偏差或信息不全导致的现实世界人类行为次优性挑战。
• 使AI智能体能够从人类-人类交互的离线数据集中学习有效的影响策略，即使这些策略未在数据中明确示范。
• 建模并适应人类潜在策略的变化，实现超越简单动作修改的长期影响。
• 证明离线强化学习可将影响策略泛化至新任务，而无需先前的成功影响示例。
• 在真实的人机协作中评估影响的有效性，展示人类行为的可测量适应。

提出的方法

• 该方法在多样化的人类-人类交互数据集上使用离线强化学习，学习影响策略，无需在线人类交互或环境模拟器。
• 通过引入潜在策略建模组件扩展离线强化学习，以估计并基于人类伙伴的潜在行为策略进行条件控制。
• 智能体从观察到的行为中推断人类策略，并相应调整其影响策略，例如通过阻止访问某些物品来引导目标选择。
• 该方法结合观察到的人类行为特征（如掉落盘子或在附近拾起盘子）来推断数据中未直接示范的有效影响策略。
• 采用记忆增强的离线强化学习架构和潜在策略推断头，以提升泛化能力和适应性。
• 该方法在Overcooked环境中进行评估，智能体学习引导人类实现更高效的任务协作。

Figure 1: Informative example of how an agent may deduce influencing strategies from offline data.

实验结果

研究问题

• RQ1 当训练数据中不存在此类策略时，离线强化学习是否仍能学习到有效的影响策略以应对次优人类行为？
• RQ2 离线强化学习智能体能否根据人类伙伴潜在行为策略的变化调整其影响策略？
• RQ3 与标准离线强化学习或行为克隆相比，建模人类潜在策略是否能带来更好的人类行为影响表现？
• RQ4 智能体的影响是否能引发人类行为的可测量、长期变化（如策略转变），而不仅仅是短暂的动作改变？
• RQ5 智能体的成功是源于对人类的影响，还是源于其自身任务性能的内在优势？

主要发现

• 所提出的潜在离线强化学习方法在影响人类行为方面显著优于标准离线强化学习和行为克隆，人类-机器人协作中的平均奖励更高。
• 记忆增强的离线强化学习和潜在离线强化学习均优于朴素的行为克隆，且潜在离线强化学习在输入信息量相同的情况下，表现明显优于记忆离线强化学习。
• 智能体成功引发人类策略转变：即使障碍被移除，人类仍会改变行为以避开被阻挡的食材，表明产生了长期影响。
• 在交互的最后100个时间步，人类伙伴的表现显著优于前100个时间步，表明人类已适应智能体的影响。
• 当与无法被影响的脚本策略配对时，所有智能体表现相似，排除了智能体成功是源于孤立任务性能而非人类影响的可能性。
• 定性分析表明，智能体学习到了自适应策略，例如仅在推断出人类不针对洋葱时才阻断其访问，体现出策略性意识。

Figure 2: Illustration of the Overcooked environment. The goal is to place three ingredients (tomatoes or onions) in a pot, plate the soup from the pot, and deliver it, as many times as possible within the time limit. The ego agent is the one in the blue hat.