[Paper Review] Choice Set Misspecification in Reward Inference

Rachel Freedman, Rohin Shah+1 more|arXiv (Cornell University)|Jan 19, 2021

Reinforcement Learning in Robotics8 references4 citations

TL;DR

This paper introduces choice set misspecification in reward inference, where a robot incorrectly assumes the set of feedback options a human could have chosen. It classifies misspecification into five distinct classes, showing that while some types cause severe performance degradation (e.g., inferring the opposite of the true reward), others are provably neutral in expectation, enabling more robust design of reward learning systems by favoring benign misspecification classes.

ABSTRACT

Specifying reward functions for robots that operate in environments without a natural reward signal can be challenging, and incorrectly specified rewards can incentivise degenerate or dangerous behavior. A promising alternative to manually specifying reward functions is to enable robots to infer them from human feedback, like demonstrations or corrections. To interpret this feedback, robots treat as approximately optimal a choice the person makes from a choice set, like the set of possible trajectories they could have demonstrated or possible corrections they could have made. In this work, we introduce the idea that the choice set itself might be difficult to specify, and analyze choice set misspecification: what happens as the robot makes incorrect assumptions about the set of choices from which the human selects their feedback. We propose a classification of different kinds of choice set misspecification, and show that these different classes lead to meaningful differences in the inferred reward and resulting performance. While we would normally expect misspecification to hurt, we find that certain kinds of misspecification are neither helpful nor harmful (in expectation). However, in other situations, misspecification can be extremely harmful, leading the robot to believe the opposite of what it should believe. We hope our results will allow for better prediction and response to the effects of misspecification in real-world reward inference.

Motivation & Objective

• To identify and formalize a new source of misspecification in reward inference: incorrect assumptions about the human's choice set of feedback options.
• To classify choice set misspecification into distinct, empirically distinguishable classes based on the relationship between human and robot choice sets and the location of the optimal feedback.
• To analyze how different classes of misspecification affect inferred rewards and resulting agent performance, including regret and belief accuracy.
• To demonstrate that not all misspecification is equally harmful, with some types being provably neutral in expectation.
• To inform the design of more robust reward learning systems by identifying strategies to minimize harmful misspecification, such as favoring larger robot choice sets.

Proposed method

• Formalizes choice set misspecification within the reward-rational implicit choice (RRiC) framework, modeling human feedback as drawn from a choice set C_H and robot belief as drawn from C_R.
• Defines a classification of five misspecification classes based on the set relationship (e.g., C_H ⊆ C_R, C_R ⊆ C_H) and the position of the optimal feedback c* relative to both sets.
• Uses Boltzmann rationality with a rationality parameter β to model human choice probabilities over feedback options.
• Employs probabilistic inference over reward parameters θ using Bayes' rule: P(θ|c_H, C_R) ∝ P(c_H|θ, C_R)P(θ), where P(c_H|θ, C_R) is computed via the RRiC model.
• Evaluates performance using regret, defined as the difference in expected reward between the true reward and the inferred reward under misspecification.
• Conducts empirical evaluations in gridworld environments with varying choice set assumptions, comparing inferred beliefs and resulting behavior under different misspecification classes.

Experimental results

Research questions

• RQ1 How does misspecification in the robot’s assumed choice set of human feedback options affect the inferred reward and resulting agent behavior?
• RQ2 Are there specific classes of choice set misspecification that lead to provably neutral effects in expectation, despite incorrect assumptions?
• RQ3 Which types of choice set misspecification lead to the most severe performance degradation, such as inferring the opposite of the true reward?
• RQ4 Can the structure of the robot’s choice set be designed to systematically reduce the risk of harmful misspecification?
• RQ5 Under what conditions does choice set misspecification not affect expected performance, and what does this imply for safe robot learning?

Key findings

• Choice set misspecification can lead to severe performance degradation, with one class (B3) causing the robot to infer the opposite of the true reward, resulting in high regret.
• In contrast, when the optimal feedback is present in both the human and robot choice sets and the robot’s set is larger (class A2), misspecification is provably neutral in expectation, causing no change in expected performance.
• The robot’s inferred belief over reward parameters can be significantly distorted when the human’s actual choice set is a strict subset of the robot’s assumed set, especially if the human’s choice is suboptimal within the robot’s expanded set.
• Misspecification leads to overconfidence in some cases (e.g., when the robot’s choice set includes many suboptimal options), and underconfidence in others, depending on the class of misspecification.
• Empirical results show that in class B3, where the human’s choice set contains options not in the robot’s set and the optimal feedback is in the robot’s set but not the human’s, the robot infers a reward that maximizes the opposite of the true reward.
• The study suggests that designing robot choice sets to be larger than the human’s (favoring class A2) can lead to more benign misspecification, reducing the risk of catastrophic inference errors.