[Paper Review] Learning Robust Rewards with Adversarial Inverse Reinforcement Learning

Justin Fu, Katie Luo+1 more|arXiv (Cornell University)|Oct 30, 2017

Receptor Mechanisms and Signaling80 citations

TL;DR

AIRL presents a scalable IRL method that learns disentangled rewards, enabling policy optimization under unknown or shifted dynamics, and outperforming prior IRL and GAN-based methods on transfer tasks while matching imitation benchmarks.

ABSTRACT

Reinforcement learning provides a powerful and general framework for decision making and control, but its application in practice is often hindered by the need for extensive feature and reward engineering. Deep reinforcement learning methods can remove the need for explicit engineering of policy or value features, but still require a manually specified reward function. Inverse reinforcement learning holds the promise of automatic reward acquisition, but has proven exceptionally difficult to apply to large, high-dimensional problems with unknown dynamics. In this work, we propose adverserial inverse reinforcement learning (AIRL), a practical and scalable inverse reinforcement learning algorithm based on an adversarial reward learning formulation. We demonstrate that AIRL is able to recover reward functions that are robust to changes in dynamics, enabling us to learn policies even under significant variation in the environment seen during training. Our experiments show that AIRL greatly outperforms prior methods in these transfer settings.

Motivation & Objective

• Motivate reward engineering bottlenecks in RL and the need for automatic reward acquisition.
• Develop a practical IRL algorithm that yields portable rewards across varying dynamics.
• Address reward shaping and ambiguity to learn disentangled rewards.
• Demonstrate scalability to continuous control with unknown dynamics and transferability of learned rewards.

Proposed method

• Adopt an adversarial IRL framework to learn rewards and value functions jointly.
• Use a single state-action discriminator with f(s,a) linked to a disentangled reward: f(s,a,s') = g_theta(s,a) + gamma h_phi(s') - h_phi(s).
• Constrain the reward component g_theta to be state-only to achieve disentanglement from dynamics.
• Incorporate a shaping term h_phi to mitigate unintended reward shaping effects.
• Train via alternating updates: discriminate expert vs. policy samples, then update the reward model and policy.
• Provide theoretical justification that the learned g_theta recovers the ground-truth reward up to a constant in certain settings.

Experimental results

Research questions

• RQ1 Can AIRL learn disentangled rewards that are robust to changes in environment dynamics?
• RQ2 Is AIRL scalable and efficient for high-dimensional continuous control tasks?
• RQ3 Does recovering disentangled rewards improve transfer to environments with different dynamics compared to prior IRL methods?

Key findings

• AIRL learns disentangled rewards that transfer across dynamic changes, outperforming naïve IRL methods in transfer settings.
• In tabular MDPs, state-only rewards recapture the ground-truth reward up to a constant, while state-action rewards yield shaped advantages.
• On continuous-control transfer tasks, AIRL with state-only rewards succeeds where policies or non-disentangled IRL methods fail to transfer under domain shifts.
• AIRL matches GAIL on standard imitation benchmarks but significantly outperforms GAIL in transfer/generalization scenarios.
• GAN-GCL struggles with trajectory-centric learning in high-dimensional tasks, while AIRL remains scalable and effective.