[论文解读] Using Machine Teaching to Investigate Human Assumptions when Teaching Reinforcement Learners

Yun‐Shiuan Chuang, Xuezhou Zhang+4 more|arXiv (Cornell University)|Sep 5, 2020

Reinforcement Learning in Robotics参考文献 32被引用 6

一句话总结

本文研究人类在探索-利用任务中，如何通过评价性反馈指导Q-learning智能体，将其视为机器教学问题。研究发现，人们以次优但有效的方式教学，偏好低折扣率（γ=0）和高学习率（α=0.9）的智能体，假设学习者具有可预测性且可即时控制，且实时反馈可视化仅对教学效率产生微弱改善。

ABSTRACT

Successful teaching requires an assumption of how the learner learns - how the learner uses experiences from the world to update their internal states. We investigate what expectations people have about a learner when they teach them in an online manner using rewards and punishment. We focus on a common reinforcement learning method, Q-learning, and examine what assumptions people have using a behavioral experiment. To do so, we first establish a normative standard, by formulating the problem as a machine teaching optimization problem. To solve the machine teaching optimization problem, we use a deep learning approximation method which simulates learners in the environment and learns to predict how feedback affects the learner's internal states. What do people assume about a learner's learning and discount rates when they teach them an idealized exploration-exploitation task? In a behavioral experiment, we find that people can teach the task to Q-learners in a relatively efficient and effective manner when the learner uses a small value for its discounting rate and a large value for its learning rate. However, they still are suboptimal. We also find that providing people with real-time updates of how possible feedback would affect the Q-learner's internal states weakly helps them teach. Our results reveal how people teach using evaluative feedback and provide guidance for how engineers should design machine agents in a manner that is intuitive for people.

研究动机与目标

• 理解人类在使用奖励与惩罚指导Q-learning智能体时的教学策略。
• 识别人类在使用评价性反馈进行在线教学时，对学习者动态所作的假设。
• 评估实时反馈可视化是否提升教学有效性。
• 构建一个机器教学框架，将最优教学建模为Q-learner的规范性基准。
• 通过使智能体参数与人类教学期望保持一致，指导设计更友好人机交互的强化学习智能体。

提出的方法

• 将教学任务建模为机器教学优化问题，将教师目标定义为最小化策略收敛所需步数。
• 使用深度学习近似逆教学过程，模拟反馈如何影响Q-learner的内部状态。
• 设计一项行为实验，让人类参与者在需要探索与利用的网格世界环境中指导Q-learner。
• 通过改变Q-learner的超参数（γ和α），测试哪些配置最易被人类教学。
• 实施白盒条件，让参与者实时查看Q-value更新，以评估其对教学效率的影响。
• 进行置换检验，评估人类教学策略是否随时间发生改变。

实验结果

研究问题

• RQ1 当通过评价性反馈教学时，人类教师对Q-learning智能体的学习率和折扣率持何种假设？
• RQ2 与最优机器教学算法相比，人类教学在教学效率方面表现如何？
• RQ3 提供关于奖励如何影响Q-value的实时反馈，是否能提升人类教学表现？
• RQ4 哪种Q-learner超参数配置最易于人类教学？
• RQ5 人类教师在教学过程中是否会随时间调整其策略？

主要发现

• 人类教师以次优但有效的方式指导Q-learner，所需步数多于最优机器教学算法（后者在11步内实现收敛）。
• 最易教学的Q-learner具有γ=0的折扣率和α=0.9的学习率，人类教师在实践中隐含地将其视为“最喜爱的学生”。
• 参与者随时间调整了教学策略，置换检验显示反馈模式非随机。
• 提供关于反馈如何影响Q-value的实时可视化仅微弱提升了教学效率，表明对内部状态的洞察收益有限。
• 能否看到智能体的内部状态（白盒 vs. 黑盒）对教学表现影响甚微。
• 人类教师假设学习者具有可预测性且可即时控制——即未来折扣极小、对奖励有强烈即时响应的智能体。