[Paper Review] Trust-Aware Decision Making for Human-Robot Collaboration: Model Learning and Planning

Min Chen, Stefanos Nikolaidis+3 more|arXiv (Cornell University)|Jan 12, 2018

Human-Automation Interaction and SafetyPsychology16 citations

TL;DR

This paper proposes a trust-aware POMDP model that enables robots to infer and influence human trust during collaboration by learning from interaction data. By strategically manipulating trust—such as intentionally failing on low-risk tasks—the robot improves long-term team performance, outperforming both myopic and trust-maximizing policies in simulated and real-world table-clearing tasks.

ABSTRACT

Trust in autonomy is essential for effective human-robot collaboration and user adoption of autonomous systems such as robot assistants. This paper introduces a computational model which integrates trust into robot decision-making. Specifically, we learn from data a partially observable Markov decision process (POMDP) with human trust as a latent variable. The trust-POMDP model provides a principled approach for the robot to (i) infer the trust of a human teammate through interaction, (ii) reason about the effect of its own actions on human trust, and (iii) choose actions that maximize team performance over the long term. We validated the model through human subject experiments on a table-clearing task in simulation (201 participants) and with a real robot (20 participants). In our studies, the robot builds human trust by manipulating low-risk objects first. Interestingly, the robot sometimes fails intentionally in order to modulate human trust and achieve the best team performance. These results show that the trust-POMDP calibrates trust to improve human-robot team performance over the long term. Further, they highlight that maximizing trust alone does not always lead to the best performance.

Motivation & Objective

• To address the challenge of misaligned human trust in autonomous systems, which can hinder effective human-robot collaboration.
• To close the loop between trust modeling and robot decision-making by integrating trust as a latent variable in a POMDP framework.
• To enable robots to infer human trust levels through interaction and adapt actions to calibrate trust to match the robot's actual capabilities.
• To improve long-term team performance by balancing trust calibration with task efficiency, especially in the presence of robot failures.
• To demonstrate that maximizing trust is not always optimal, and that strategic trust modulation can lead to better overall performance.

Proposed method

• The trust-POMDP model represents trust as a latent variable in a partially observable Markov decision process, enabling the robot to reason about unobserved human trust levels.
• The model learns a trust dynamics component from human interaction data, capturing how trust evolves in response to robot actions.
• A human decision model is learned to link trust levels to human intervention behavior, such as stopping the robot from performing high-risk actions.
• The robot uses the POMDP policy to select actions that maximize long-term team performance, balancing trust building and task progress.
• In scenarios with high failure probability, the robot intentionally fails on low-risk objects (e.g., bottles) to reduce overconfidence and prevent later failures on high-value objects (e.g., glass).
• The model is trained and validated using data from 201 participants in simulation and 20 participants with a real robot in a table-clearing task.

Experimental results

Research questions

• RQ1 How can a robot systematically infer human trust during human-robot collaboration when trust is not directly observable?
• RQ2 What is the impact of robot actions on the evolution of human trust, and how can this be modeled computationally?
• RQ3 Can a robot improve long-term team performance by strategically modulating human trust, even if it means intentional failures?
• RQ4 Is maximizing human trust always optimal for task performance, especially when robot failures are likely?
• RQ5 How does the performance of a trust-aware policy compare to a myopic policy that ignores trust in action selection?

Key findings

• The trust-POMDP strategy significantly reduced the human intervention rate compared to the myopic strategy, indicating improved collaboration and performance.
• The robot successfully built trust by first handling low-risk objects (e.g., plastic bottles), which led to fewer human interventions during high-risk actions.
• In scenarios with high failure probability on the wine glass, the trust-POMDP policy intentionally failed on bottles to reduce overconfidence, preventing later failures.
• The performance-maximizing policy achieved a mean accumulated reward of -1.36 over 10,000 runs, significantly better than the trust-maximizing policy’s -1.65 (p < 0.001).
• Maximizing trust was suboptimal in the presence of robot failures, as it led to steeper trust drops when failures occurred, reducing long-term performance.
• The model demonstrated that trust calibration—neither over- nor under-trust—is essential for optimal human-robot team performance.