[Paper Review] Active Learning for Risk-Sensitive Inverse Reinforcement Learning

Rui Chen, Wenshuo Wang+2 more|arXiv (Cornell University)|Sep 14, 2019

Reinforcement Learning in Robotics22 references4 citations

TL;DR

This paper proposes an active learning framework for risk-sensitive inverse reinforcement learning (RS-IRL) that adaptively selects informative disturbances to accelerate convergence and reduce variance in learning human risk preferences. By computing future refinement directions on the probability simplex, the method identifies high-impact demonstrations without solving optimization problems, significantly improving sample efficiency in both single- and multi-step settings.

ABSTRACT

One typical assumption in inverse reinforcement learning (IRL) is that human experts act to optimize the expected utility of a stochastic cost with a fixed distribution. This assumption deviates from actual human behaviors under ambiguity. Risk-sensitive inverse reinforcement learning (RS-IRL) bridges such gap by assuming that humans act according to a random cost with respect to a set of subjectively distorted distributions instead of a fixed one. Such assumption provides the additional flexibility to model human's risk preferences, represented by a risk envelope, in safe-critical tasks. However, like other learning from demonstration techniques, RS-IRL could also suffer inefficient learning due to redundant demonstrations. Inspired by the concept of active learning, this research derives a probabilistic disturbance sampling scheme to enable an RS-IRL agent to query expert support that is likely to expose unrevealed boundaries of the expert's risk envelope. Experimental results confirm that our approach accelerates the convergence of RS-IRL algorithms with lower variance while still guaranteeing unbiased convergence.

Motivation & Objective

• To address slow convergence in RS-IRL due to redundant expert demonstrations.
• To model human risk preferences under ambiguity using coherent risk measures and a risk envelope.
• To develop an active learning scheme that selects disturbances likely to expose unrevealed boundaries of the expert’s risk envelope.
• To enable efficient, adaptive query of expert demonstrations without solving full optimization problems at each step.
• To validate the method in a simulated car-following task with risk-seeking and risk-averse participants.

Proposed method

• The method derives a probabilistic disturbance sampling scheme that estimates future refinement directions on the probability simplex to identify informative disturbances.
• It uses quadratic feature functions to model cost functions, including relative distance, velocity, and control effort penalties.
• The risk envelope is inferred via half-space constraints from expert demonstrations, with active learning prioritizing disturbances that most likely refine the envelope boundary.
• A disturbance realization is evaluated by computing its potential to generate a new half-space constraint, avoiding full optimization.
• The approach is generalized to multi-step settings by analyzing disturbances across sequential stages in a continuous episode.
• K-means clustering is used to discretize expert reaction sequences into 15 representative behaviors for simulation and analysis.

Experimental results

Research questions

• RQ1 Can active learning improve the sample efficiency of risk-sensitive inverse reinforcement learning by selecting informative demonstrations?
• RQ2 How can an RS-IRL agent identify disturbances that are most likely to refine the boundaries of the expert’s risk envelope?
• RQ3 Does the proposed active learning scheme reduce variance and accelerate convergence compared to random or uniform demonstration sampling?
• RQ4 Can the method generalize to multi-step sequential decision-making tasks such as autonomous driving?
• RQ5 How do risk preferences, such as risk-seeking or risk-averse behavior, influence the selection of informative disturbances?

Key findings

• The active learning scheme significantly accelerates convergence of RS-IRL with lower variance across repeated episodes compared to baseline methods.
• The method identifies disturbances that lead to successful risk envelope refinements, with stage 25 in the simulation producing the dominant constraint for a risk-seeking participant.
• At stage 25, the active learning sampling probability was 0.4227 for the acceleration maneuver, aligning with the expert’s risk preference.
• The disturbance sampling probability distribution p = [0.3802, 0.4227, 0.1971] at stage 25 favored the maneuver that best exposed the expert’s risk envelope.
• The approach enables efficient inference of the risk envelope without solving full optimization problems at each step, relying instead on refinement direction estimation.
• Experimental results confirm unbiased convergence while reducing redundant demonstrations, validating the method’s effectiveness in safe-critical settings.