[Paper Review] Imitating Interactive Intelligence

Josh Abramson, Arun Ahuja+27 more|arXiv (Cornell University)|Dec 10, 2020

Multimodal Machine Learning Applications69 references43 citations

TL;DR

The paper proposes training interactive agents in a 3D virtual Playroom by imitation learning from human-human data, augmented with auxiliary losses and evaluation models to approximate human evaluation and enable generalization beyond the training data.

ABSTRACT

A common vision from science fiction is that robots will one day inhabit our physical spaces, sense the world as we do, assist our physical labours, and communicate with us through natural language. Here we study how to design artificial agents that can interact naturally with humans using the simplification of a virtual environment. This setting nevertheless integrates a number of the central challenges of artificial intelligence (AI) research: complex visual perception and goal-directed physical control, grounded language comprehension and production, and multi-agent social interaction. To build agents that can robustly interact with humans, we would ideally train them while they interact with humans. However, this is presently impractical. Therefore, we approximate the role of the human with another learned agent, and use ideas from inverse reinforcement learning to reduce the disparities between human-human and agent-agent interactive behaviour. Rigorously evaluating our agents poses a great challenge, so we develop a variety of behavioural tests, including evaluation by humans who watch videos of agents or interact directly with them. These evaluations convincingly demonstrate that interactive training and auxiliary losses improve agent behaviour beyond what is achieved by supervised learning of actions alone. Further, we demonstrate that agent capabilities generalise beyond literal experiences in the dataset. Finally, we train evaluation models whose ratings of agents agree well with human judgement, thus permitting the evaluation of new agent models without additional effort. Taken together, our results in this virtual environment provide evidence that large-scale human behavioural imitation is a promising tool to create intelligent, interactive agents, and the challenge of reliably evaluating such agents is possible to surmount.

Motivation & Objective

• Unify artificial intelligence with human-like interaction in a grounded, interactive setting.
• Develop a large-scale behavioural prior by imitating human demonstrations of interaction.
• Demonstrate that interactive training improves agent behavior beyond supervised action learning.
• Show that learned agents generalize to novel states beyond exact training experiences.
• Create evaluation models whose judgments align with human evaluation to assess new agents.

Proposed method

• Use a 3D Unity-based Playroom with a mobile manipulator for perception, action, and language tasks.
• Collect a large-scale dataset of human setter-solver interactions via language games to train and evaluate imitation learners (≈610k episodes).
• Employ an action space with continuous mouse-look and keyboard controls, modeled by an autoregressive policy conditioned on multi-modal observations.
• Architect agents with a ResNet-based vision module, a multi-modal transformer, an LSTM, and separate policies for motor and language output.
• Implement behavioural cloning as the base imitation objective, augmented with auxiliary losses (Language Matching and Object-in-View) to regularize representations.
• Discuss the role of inverse reinforcement learning to address distribution mismatch and improve learning from demonstrations.

Experimental results

Research questions

• RQ1 Can large-scale human behavioural imitation yield intelligent, interactive agents in a virtual environment?
• RQ2 Do auxiliary learning signals and behavioural priors improve imitation-learning agents beyond pure BC?
• RQ3 How well do learned agents generalize to states not explicitly seen in the dataset?
• RQ4 Can evaluation models be trained to align with human judgment for scalable agent assessment?

Key findings

• Interactive training plus auxiliary losses improve agent behaviour beyond supervised learning of actions alone.
• Agents generalize to novel states and tasks not explicitly present in the training data.
• A large-scale behavioural prior trained from human demonstrations helps drive human-like responses during interaction.
• Evaluation models trained to predict human judgments align well with human assessments for new agents.