[Paper Review] When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humans

Minae Kwon, Erdem Bıyık+4 more|arXiv (Cornell University)|Jan 13, 2020

Decision-Making and Behavioral Economics46 references4 citations

TL;DR

This paper proposes a Risk-Aware human model based on Cumulative Prospect Theory to improve human-robot collaboration under uncertainty. By replacing traditional Noisy Rational models with a risk-sensitive framework, robots better predict suboptimal human behavior in risky scenarios, leading to safer and more efficient interactions in autonomous driving and cup-stacking tasks.

ABSTRACT

In order to collaborate safely and efficiently, robots need to anticipate how their human partners will behave. Some of today's robots model humans as if they were also robots, and assume users are always optimal. Other robots account for human limitations, and relax this assumption so that the human is noisily rational. Both of these models make sense when the human receives deterministic rewards: i.e., gaining either $100 or $130 with certainty. But in real world scenarios, rewards are rarely deterministic. Instead, we must make choices subject to risk and uncertainty--and in these settings, humans exhibit a cognitive bias towards suboptimal behavior. For example, when deciding between gaining $100 with certainty or $130 only 80% of the time, people tend to make the risk-averse choice--even though it leads to a lower expected gain! In this paper, we adopt a well-known Risk-Aware human model from behavioral economics called Cumulative Prospect Theory and enable robots to leverage this model during human-robot interaction (HRI). In our user studies, we offer supporting evidence that the Risk-Aware model more accurately predicts suboptimal human behavior. We find that this increased modeling accuracy results in safer and more efficient human-robot collaboration. Overall, we extend existing rational human models so that collaborative robots can anticipate and plan around suboptimal human behavior during HRI.

Motivation & Objective

• To address the limitation of existing robots that assume humans are either fully optimal or noisily rational, which fails under real-world risk and uncertainty.
• To model human decision-making as risk-aware rather than purely rational, reflecting cognitive biases like risk aversion or seeking under uncertainty.
• To improve robot planning and collaboration by integrating behavioral economics principles into human-robot interaction (HRI) models.
• To empirically validate that Risk-Aware models outperform Noisy Rational models in predicting human behavior and enhancing collaboration outcomes.

Proposed method

• Adopt Cumulative Prospect Theory (CPT) as the core human decision model, which captures nonlinear weighting of probabilities and outcomes.
• Formalize a theory-of-mind (ToM) framework where the robot models the human’s risk-sensitive preferences based on perceived rewards and probabilities.
• Integrate CPT into a probabilistic inference system to predict human actions under risk, replacing the standard expected utility maximization with a prospect value function.
• Train and compare two robot models: a Noisy Rational baseline and a Risk-Aware model using the same human demonstration data.
• Use the predicted human behavior to guide robot planning, minimizing interference and improving task efficiency in collaborative settings.
• Conduct user studies in simulated autonomous driving and physical cup-stacking tasks to evaluate model accuracy and collaboration quality.

Experimental results

Research questions

• RQ1 Can a Risk-Aware human model based on Cumulative Prospect Theory more accurately predict human behavior in uncertain, high-stakes scenarios than a Noisy Rational model?
• RQ2 In what types of decision scenarios is risk-aware modeling most critical for improving robot prediction and collaboration?
• RQ3 Does using a Risk-Aware model lead to safer and more efficient human-robot collaboration compared to traditional rational models?
• RQ4 How do human users perceive and respond to robots that anticipate their risk-averse or risk-seeking tendencies?

Key findings

• The Risk-Aware model, based on Cumulative Prospect Theory, significantly outperformed the Noisy Rational baseline in predicting human behavior in both autonomous driving simulations and physical cup-stacking tasks.
• In the cup-stacking task, participants completed the collaboration faster and reported less interference when interacting with the Risk-Aware robot compared to the Noisy Rational one.
• Participants subjectively preferred the Risk-Aware robot, reporting it was more efficient and better aligned with their intentions, with a marginal but positive preference (p < .07).
• Users reported that the Noisy Rational robot often interfered by attempting to pick up the same cup they were reaching for, increasing task time.
• The Risk-Aware robot reduced interference by anticipating that humans would avoid risky actions, such as grabbing unstable cups.
• In a grid-world experiment with longer horizons, the Risk-Aware model maintained higher accuracy in predicting sequences of human actions than the Noisy Rational model.