[Paper Review] Risk-Aware Active Inverse Reinforcement Learning

Daniel S. Brown, Yuchen Cui+1 more|arXiv (Cornell University)|Jan 8, 2019

Machine Learning and AlgorithmsComputer Science23 references19 citations

TL;DR

This paper proposes Risk-Aware Active Inverse Reinforcement Learning (IRL), which selects queries based on the potential for high generalization error in the robot's policy, using Value-at-Risk (VaR) bounds to prioritize states where performance loss is most likely. It outperforms entropy-based and information-gain-based methods in sample efficiency and provides a tight, computable performance upper bound for safe decision-making.

ABSTRACT

Active learning from demonstration allows a robot to query a human for specific types of input to achieve efficient learning. Existing work has explored a variety of active query strategies; however, to our knowledge, none of these strategies directly minimize the performance risk of the policy the robot is learning. Utilizing recent advances in performance bounds for inverse reinforcement learning, we propose a risk-aware active inverse reinforcement learning algorithm that focuses active queries on areas of the state space with the potential for large generalization error. We show that risk-aware active learning outperforms standard active IRL approaches on gridworld, simulated driving, and table setting tasks, while also providing a performance-based stopping criterion that allows a robot to know when it has received enough demonstrations to safely perform a task.

Motivation & Objective

• To address the lack of performance-risk awareness in existing active IRL methods, which focus on uncertainty reduction rather than actual policy risk.
• To enable robots to actively query humans in states where their learned policy may generalize poorly under the true reward function.
• To provide a performance-based stopping criterion that signals when enough demonstrations have been collected for safe task execution.
• To leverage recent advances in tight performance bounds for IRL to make active learning decisions based on actual policy risk rather than uncertainty.

Proposed method

• The method uses Bayesian IRL to maintain a posterior distribution over reward functions from demonstrated trajectories.
• It computes the Value-at-Risk (VaR) of policy loss at each candidate query configuration by estimating the worst-case performance across the posterior distribution of reward functions.
• The robot selects the query configuration with the highest α-VaR (e.g., 0.95-VaR) to minimize the risk of large generalization error.
• It employs MCMC sampling to estimate the posterior over reward functions and computes placement loss relative to the MAP reward function to determine VaR.
• The approach integrates a stopping condition based on whether the 0.95-VaR bound falls below a predefined threshold, indicating safe policy deployment.
• The method is evaluated on gridworld, simulated driving, and real-robot table-setting tasks with synthetic demonstrations and real-world validation.

Experimental results

Research questions

• RQ1 Can active IRL be designed to prioritize queries based on the risk of policy generalization error rather than uncertainty or information gain?
• RQ2 Can VaR-based performance bounds provide a tight and meaningful upper bound on policy loss when the true reward is unknown?
• RQ3 Does risk-aware query selection lead to faster convergence and lower generalization error compared to standard active IRL methods?
• RQ4 Can the VaR bound serve as a reliable stopping criterion for active learning in IRL?

Key findings

• Risk-aware active IRL achieved significantly lower mean placement error than random and entropy-based query strategies in table-setting tasks, with a 30-50% reduction in error over 10 demonstrations.
• The 0.95-VaR upper bound accurately and tightly bounded the actual worst-case placement loss across all experiments, becoming progressively tighter with more demonstrations.
• The method required three orders of magnitude less computation than information-gain-based active IRL while achieving comparable or better performance.
• The approach enabled a meaningful stopping condition: when the 0.95-VaR bound dropped below a threshold, the robot could safely perform the task without further queries.
• In simulated driving and gridworld tasks, risk-aware queries reduced policy generalization error more effectively than entropy-based queries, especially in high-uncertainty regions.
• Real-robot validation confirmed that policies learned via risk-aware queries generalized better to novel test configurations than those from random or entropy-based queries.