[Paper Review] One-Shot Imitation Learning

Yan Duan, Marcin Andrychowicz+6 more|arXiv (Cornell University)|Mar 21, 2017

Domain Adaptation and Few-Shot LearningComputer Science40 references227 citations

TL;DR

The paper introduces a meta-learning approach for one-shot imitation learning, enabling a neural policy to imitate a new task from a single demonstration by conditioning on that demonstration and using soft attention to generalize across unseen tasks.

ABSTRACT

Imitation learning has been commonly applied to solve different tasks in isolation. This usually requires either careful feature engineering, or a significant number of samples. This is far from what we desire: ideally, robots should be able to learn from very few demonstrations of any given task, and instantly generalize to new situations of the same task, without requiring task-specific engineering. In this paper, we propose a meta-learning framework for achieving such capability, which we call one-shot imitation learning. Specifically, we consider the setting where there is a very large set of tasks, and each task has many instantiations. For example, a task could be to stack all blocks on a table into a single tower, another task could be to place all blocks on a table into two-block towers, etc. In each case, different instances of the task would consist of different sets of blocks with different initial states. At training time, our algorithm is presented with pairs of demonstrations for a subset of all tasks. A neural net is trained that takes as input one demonstration and the current state (which initially is the initial state of the other demonstration of the pair), and outputs an action with the goal that the resulting sequence of states and actions matches as closely as possible with the second demonstration. At test time, a demonstration of a single instance of a new task is presented, and the neural net is expected to perform well on new instances of this new task. The use of soft attention allows the model to generalize to conditions and tasks unseen in the training data. We anticipate that by training this model on a much greater variety of tasks and settings, we will obtain a general system that can turn any demonstrations into robust policies that can accomplish an overwhelming variety of tasks. Videos available at https://bit.ly/nips2017-oneshot .

Motivation & Objective

• Enable a policy to learn new tasks from one demonstration within a potentially infinite task distribution.
• Develop a training framework where the policy maps (demonstration, current observation) to actions for unseen tasks.
• Show that attention mechanisms enable generalization across varying task configurations and numbers of objects.

Proposed method

• Formulate a policy pi(a|o, d) conditioned on an input demonstration d and current observation o.
• Train with demonstrations from a distribution of tasks so that one demonstration can guide actions on new instances of the same task.
• Use temporal dropout to downsample long demonstrations and improve generalization.
• Apply neighborhood attention over block positions to relate blocks and extract relevant contextual information.
• Employ a three-module architecture: Demonstration Network, Context Network, and Manipulation Network.
• Utilize soft attention (and multi-head attention) to process variable-length demonstrations and variable object counts.

Experimental results

Research questions

• RQ1 Can a single demonstration of a new task enable robust policy execution on unseen instances of that task?
• RQ2 Does conditioning on the full demonstration outperform conditioning on just the final state or on a limited snapshot of the trajectory?
• RQ3 Does training with behavioral cloning equal or rival DAGGER in this one-shot imitation setting?
• RQ4 To what extent can the model generalize to tasks not seen during training within a block-stacking domain?

Key findings

• The one-shot imitation approach enables the policy to perform well on novel task instances after one demonstration.
• Conditioning on the entire demonstration starts to outperform final-state conditioning as task difficulty (stages) increases.
• Temporal dropout with demonstration downsampling improves generalization and acts as a regularizer.
• Behavioral cloning performs comparably to DAGGER in this setting, suggesting interactive supervision may not be necessary.
• Attention visualizations show the model focusing on a small subset of blocks and key frames corresponding to task stages.