[Paper Review] From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following

Justin Fu, Anoop Korattikara+2 more|arXiv (Cornell University)|Feb 20, 2019

Multimodal Machine Learning ApplicationsComputer Science66 citations

TL;DR

The paper proposes Language-Conditioned Reward Learning (LC-RL), grounding natural language commands as transferable reward functions via MaxEnt IRL, and shows rewards generalize to new tasks and scenes while language-conditioned policies underperform in generalization.

ABSTRACT

Reinforcement learning is a promising framework for solving control problems, but its use in practical situations is hampered by the fact that reward functions are often difficult to engineer. Specifying goals and tasks for autonomous machines, such as robots, is a significant challenge: conventionally, reward functions and goal states have been used to communicate objectives. But people can communicate objectives to each other simply by describing or demonstrating them. How can we build learning algorithms that will allow us to tell machines what we want them to do? In this work, we investigate the problem of grounding language commands as reward functions using inverse reinforcement learning, and argue that language-conditioned rewards are more transferable than language-conditioned policies to new environments. We propose language-conditioned reward learning (LC-RL), which grounds language commands as a reward function represented by a deep neural network. We demonstrate that our model learns rewards that transfer to novel tasks and environments on realistic, high-dimensional visual environments with natural language commands, whereas directly learning a language-conditioned policy leads to poor performance.

Motivation & Objective

• Motivate grounding natural language commands as reward functions instead of direct policy conditioning.
• Develop a scalable method to learn language-conditioned rewards that generalize across tasks and environments.
• Evaluate whether reward-grounded approaches transfer better than language-conditioned policies to novel scenes and tasks.

Proposed method

• Adopt MaxEnt IRL to learn a language-conditioned reward function r(o, a, L) shared across multiple tasks.
• Represent rewards with a neural network taking panoramic image observations and language input; language via LSTM embedding, image via CNN with shared weights across views.
• Train using exact IRL gradient updates with known dynamics (via dynamic programming) to match expert trajectory distributions across tasks.
• Use multi-task setup where tasks share observation/action spaces and ground language L as a task context, enabling cross-task transfer.
• Compare LC-RL to policy-based baselines (optimal policy cloning, AGILE, GAIL variants) and to an oracle reward regression baseline.

Experimental results

Research questions

• RQ1 Can language-conditioned rewards learned by IRL transfer to novel tasks and unseen environments more effectively than language-conditioned policies?
• RQ2 Does grounding language as a reward function enable robust generalization across different scenes and task configurations in vision-based instruction following?
• RQ3 How does LC-RL compare to policy-based methods and to oracle reward regression in terms of generalization performance?
• RQ4 What are the practical limitations and trade-offs of exact IRL training when dynamics are known during training but not at test time?

Key findings

• LC-RL achieves strong generalization to new tasks and unseen houses, outperforming policy-based baselines in test settings.
• Reward regression (oracle rewards) yields higher absolute performance, illustrating the gap between learned rewards and ground-truth rewards, but LC-RL remains competitive without oracle supervision.
• On the SUNCG indoor navigation/pick-and-place tasks, LC-RL achieves 66.9% total success on Test-Task and 51.9% on Test-House, outperforming GAIL-Exact and AGILE baselines in several settings.
• Policy cloning performs worse in both training and testing scenarios, highlighting the difficulty of zero-shot policy transfer for language-conditioned tasks.
• Re-optimizing learned rewards with DQN is challenging; exact solvers (Q-iteration) provide substantially stronger performance than learned rewards with model-free RL, though reward regression can still benefit from shaping when dynamics are known.