[Paper Review] Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text

Felix Hill, Soňa Mokrá+2 more|arXiv (Cornell University)|May 19, 2020

Multimodal Machine Learning Applications43 references32 citations

TL;DR

The paper introduces SHIFTT, a method that uses pretrained text encoders (e.g., BERT) to bootstrap deep RL agents, enabling zero-shot transfer from template-based instructions to natural human instructions in 3D object manipulation tasks.

ABSTRACT

Recent work has described neural-network-based agents that are trained with reinforcement learning (RL) to execute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users. However, the optimisation of multi-goal motor policies via deep RL from scratch requires many episodes of experience. Consequently, instruction-following with deep RL typically involves language generated from templates (by an environment simulator), which does not reflect the varied or ambiguous expressions of real users. Here, we propose a conceptually simple method for training instruction-following agents with deep RL that are robust to natural human instructions. By applying our method with a state-of-the-art pre-trained text-based language model (BERT), on tasks requiring agents to identify and position everyday objects relative to other objects in a naturalistic 3D simulated room, we demonstrate substantially-above-chance zero-shot transfer from synthetic template commands to natural instructions given by humans. Our approach is a general recipe for training any deep RL-based system to interface with human users, and bridges the gap between two research directions of notable recent success: agent-centric motor behavior and text-based representation learning.

Motivation & Objective

• Motivate robust instruction-following by reducing reliance on template-generated language.
• Propose a simple transfer-learning recipe from text encoders to RL policies.
• Demonstrate zero-shot generalization to natural human instructions in a 3D room with ShapeNet objects.
• Analyze how different language encoders and training augmentations affect generalization and robustness.

Proposed method

• Use a standard RL architecture with visual processing, language embeddings, memory, and action/value heads.
• Incorporate pretrained language encoders (BERT) in multiple encoding schemes (mean-pooled, self-attention, cross-modal attention).
• Freeze BERT weights to avoid overfitting, and optionally add a learned self-attention layer or cross-modal self-attention to fuse vision and language.
• Train the agent in simulation with synthetic template instructions, using SHIFTT steps to distribute semantic knowledge into policy parameters.
• Evaluate zero-shot performance on natural human instructions and synonyms collected via crowd-sourcing, without retraining on human data.
• Examine the impact of typo noise and tokenization (WordPiece) on robustness to human instructions.

Experimental results

Research questions

• RQ1 Can pretrained language encoders enable zero-shot transfer from template instructions to natural human instructions in RL agents?
• RQ2 Which language-encoder architectures (mean-pooled BERT, self-attention, cross-modal attention) best support grounding language in vision and action?
• RQ3 How does typo noise and WordPiece tokenization affect robustness to human instructions?
• RQ4 To what extent can agents trained in simulation with synthetic language perform on natural, varied human instructions?
• RQ5 Does the SHIFTT approach scale to tasks requiring object identification and object placement relations in realistic environments?

Key findings

• Pretrained text encoders substantially improve accuracy on both reference and putting tasks, compared to non-pretrained baselines.
• BERT-based encoders with mean pooling achieved up to 77% (synonym evaluation) and 94% (reference-like evaluation) accuracies in the respective tests.
• Contextual representations (BERT) outperform context-independent embeddings and WordPiece alone in generalization to natural instructions.
• Cross-modal self-attention plus typo-noise yields strong robustness to human instructions, particularly for natural placing instructions (e.g., 70% accuracy in natural instruction tests).
• WordPiece tokenization and tuned self-attention layers provide robustness when combined with typo noise, though overfitting risks are mitigated by freezing BERT weights.