[Paper Review] Here's What I've Learned: Asking Questions that Reveal Reward Learning

Soheil Habibian, Ananth Jonnavittula+1 more|arXiv (Cornell University)|Jul 2, 2021

Reinforcement Learning in Robotics48 references4 citations

TL;DR

This paper proposes a dual-objective framework for active preference learning in human-robot interaction, where robots balance asking informative questions (to learn efficiently) with revealing questions (to transparently communicate their knowledge to humans). By modeling human perception of robot questions, the approach enables robots to learn as quickly as state-of-the-art methods while making their learning progress visibly clear, leading to faster human trust and deployment decisions.

ABSTRACT

Robots can learn from humans by asking questions. In these questions the robot demonstrates a few different behaviors and asks the human for their favorite. But how should robots choose which questions to ask? Today's robots optimize for informative questions that actively probe the human's preferences as efficiently as possible. But while informative questions make sense from the robot's perspective, human onlookers often find them arbitrary and misleading. In this paper we formalize active preference-based learning from the human's perspective. We hypothesize that -- from the human's point-of-view -- the robot's questions reveal what the robot has and has not learned. Our insight enables robots to use questions to make their learning process transparent to the human operator. We develop and test a model that robots can leverage to relate the questions they ask to the information these questions reveal. We then introduce a trade-off between informative and revealing questions that considers both human and robot perspectives: a robot that optimizes for this trade-off actively gathers information from the human while simultaneously keeping the human up to date with what it has learned. We evaluate our approach across simulations, online surveys, and in-person user studies. Videos of our user studies and results are available here: https://youtu.be/tC6y_jHN7Vw.

Motivation & Objective

• To address the gap in human-robot interaction where informative robot questions can mislead users about the robot's knowledge state.
• To formalize how robot questions reveal information about the robot's current understanding to human observers.
• To develop a method that balances robot-centric information gain with human-centric transparency in active learning.
• To evaluate whether revealing questions improve human perception of robot competence and readiness to deploy.
• To enable robots to communicate their learning progress explicitly, enhancing trust and interpretability in human-robot collaboration.

Proposed method

• Formalize a cognitive human model that maps robot questions to perceived knowledge states, capturing how humans infer what the robot knows from its questions.
• Introduce a continuous trade-off between 'informative' (maximizing information gain for the robot) and 'revealing' (maximizing transparency to the human) objectives.
• Use a probabilistic reward learning framework where the robot maintains a belief over the human's reward function and selects questions that optimize the dual objective.
• Design question pairs that emphasize known behaviors (e.g., correct stacking) while varying uncertain aspects (e.g., carrying height), to make the robot’s uncertainty visible.
• Implement and evaluate the method in simulations, online surveys, and in-person user studies using a dish-stacking task.
• Use human feedback to calibrate the model and validate that participants can correctly infer the robot’s knowledge state from its questions.

Experimental results

Research questions

• RQ1 How do robot questions influence human perception of the robot’s knowledge state during active learning?
• RQ2 Can a robot’s question selection strategy be designed to simultaneously maximize information gain and transparency to the human?
• RQ3 Does making robot learning visible through question design improve human trust and decision-making about robot deployment?
• RQ4 How does the trade-off between informative and revealing questions affect learning speed and human interpretability?
• RQ5 To what extent can humans accurately infer what a robot knows based solely on the questions it asks?

Key findings

• Robots using the proposed informative + revealing trade-off learned as quickly as state-of-the-art informative-only baselines, with no significant loss in learning speed.
• Participants in user studies were able to correctly identify what the robot knew and did not know based solely on the questions it asked, validating the model’s interpretability.
• In the 'Ready to Deploy' assessment, participants deployed the robot after approximately six questions when using the revealing approach, whereas they never deployed the robot in the purely informative condition due to uncertainty about its knowledge state.
• The 'Informative' condition led to participants perceiving questions as arbitrary and misleading, reducing trust and delaying deployment decisions.
• The 'Revealing' condition enabled participants to recognize when the robot had learned the correct behavior, particularly by observing consistent correct actions (e.g., stacking) across question pairs.
• Online surveys confirmed that participants perceived the robot’s questions as more transparent and interpretable when they revealed learning progress, even when multiple task aspects were varied.