[Paper Review] Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement

Benjamin Eysenbach, Xinyang Geng+2 more|arXiv (Cornell University)|Feb 25, 2020

Reinforcement Learning in RoboticsComputer Science46 references30 citations

TL;DR

This paper shows that hindsight relabeling is inverse reinforcement learning and proposes HIPI, a framework that relabels past experience via inverse RL to accelerate learning across multiple tasks, via off-policy RL (HIPI-RL) or behavior cloning (HIPI-BC).

ABSTRACT

Multi-task reinforcement learning (RL) aims to simultaneously learn policies for solving many tasks. Several prior works have found that relabeling past experience with different reward functions can improve sample efficiency. Relabeling methods typically ask: if, in hindsight, we assume that our experience was optimal for some task, for what task was it optimal? In this paper, we show that hindsight relabeling is inverse RL, an observation that suggests that we can use inverse RL in tandem for RL algorithms to efficiently solve many tasks. We use this idea to generalize goal-relabeling techniques from prior work to arbitrary classes of tasks. Our experiments confirm that relabeling data using inverse RL accelerates learning in general multi-task settings, including goal-reaching, domains with discrete sets of rewards, and those with linear reward functions.

Motivation & Objective

• Motivate improving sample efficiency in multi-task RL by reusing past experience across arbitrarily defined tasks.
• Formalize hindsight relabeling as inverse RL and connect MaxEnt RL and MaxEnt inverse RL as dual problems.
• Develop practical HIPI algorithms that relabel trajectories or transitions and leverage them to learn policies across task families.
• Demonstrate applicability to goal-reaching and general reward structures including discrete and linear reward tasks.

Proposed method

• Frame relabeling as computing the MaxEnt inverse RL posterior p(ψ|τ) and using it to relabel trajectories.
• Derive that the optimal relabeling distribution q(ψ|τ) ∝ p(ψ) exp(Σt rψ(st,at) − log Z(ψ)).
• Show that MaxEnt RL and MaxEnt inverse RL optimize the same multi-task objective but with respect to trajectories or tasks respectively.
• Specialize to goal relabeling as a degenerate inverse RL case where rψ is infinite when the final state ≠ ψ and 0 otherwise.
• Introduce two HIPI variants: HIPI-RL (inverse RL relabeling plus off-policy RL) and HIPI-BC (inverse RL relabeling plus task-conditioned behavior cloning).
• Algorithmic details include computing partition function Z(ψ) and using a softmax sample of ψ conditioned on trajectory data.

Experimental results

Research questions

• RQ1 Can inverse RL-based relabeling improve sample efficiency for a broad class of multi-task RL problems beyond goal-reaching tasks?
• RQ2 How does relabeling with inverse RL compare to prior relabeling strategies (e.g., HER) across diverse task distributions, including linear and discrete rewards?
• RQ3 Can the combination of inverse RL relabeling with off-policy RL or behavior cloning reliably accelerate learning on complex locomotion and manipulation tasks?
• RQ4 What is the role of the partition function Z(ψ) in stabilizing relabeling across tasks with varying reward scales?

Key findings

• Relabeling with inverse RL accelerates learning across goal-reaching, discrete reward, and linear reward tasks.
• Two practical HIPI variants (HIPI-RL and HIPI-BC) outperform baselines on multiple simulated robotics domains.
• The partition function normalization is crucial; without it, relabeling can misassign trajectories to easier tasks and hinder learning.
• Relabeling improves asymptotic performance and sample efficiency across a range of task distributions, including general reward structures.
• Goal relabeling is a special case of inverse RL, enabling extension to arbitrary task families beyond goals.