[Paper Review] Gated-Attention Architectures for Task-Oriented Language Grounding

Devendra Singh Chaplot, Kanthashree Mysore Sathyendra+3 more|arXiv (Cornell University)|Jun 22, 2017

Multimodal Machine Learning Applications99 citations

TL;DR

Introduces end-to-end Gated-Attention multimodal fusion for grounding natural language in 3D environments and learns policies via RL and IL. GA units outperform concatenation in multitask and zero-shot generalization.

ABSTRACT

To perform tasks specified by natural language instructions, autonomous agents need to extract semantically meaningful representations of language and map it to visual elements and actions in the environment. This problem is called task-oriented language grounding. We propose an end-to-end trainable neural architecture for task-oriented language grounding in 3D environments which assumes no prior linguistic or perceptual knowledge and requires only raw pixels from the environment and the natural language instruction as input. The proposed model combines the image and text representations using a Gated-Attention mechanism and learns a policy to execute the natural language instruction using standard reinforcement and imitation learning methods. We show the effectiveness of the proposed model on unseen instructions as well as unseen maps, both quantitatively and qualitatively. We also introduce a novel environment based on a 3D game engine to simulate the challenges of task-oriented language grounding over a rich set of instructions and environment states.

Motivation & Objective

• Develop an end-to-end architecture for task-oriented language grounding using raw pixel input and natural language instructions.
• Propose a novel Gated-Attention fusion mechanism to combine visual and linguistic representations.
• Train policies via reinforcement learning and imitation learning to execute instructions in 3D environments.
• Demonstrate generalization to unseen instructions and unseen maps in a ViZDoom-based Doom-like setting.

Proposed method

• Process image through a CNN to obtain x_I and instruction through a GRU to obtain x_L.
• Fuse modalities with a novel Gated-Attention unit M_GA(x_I, x_L) that gates convolutional feature maps via a sigmoid-attention vector derived from x_L.
• Compare GA fusion to a baseline concatenation fusion M_concat(x_I, x_L).
• Train policies using A3C (reinforcement learning) with entropy regularization and Generalized Advantage Estimation; or Behavioral Cloning/DAgger for imitation learning.
• Utilize a Doom-based ViZDoom environment with first-person view and a 70-instruction set to assess multitask and zero-shot generalization.

Experimental results

Research questions

• RQ1 Can a gated-attention multimodal fusion improve grounding of natural language to visual elements in a 3D environment?
• RQ2 Does GA fusion enable better generalization to unseen instructions and unseen maps compared to concatenation?
• RQ3 How do reinforcement learning and imitation learning compare in this task setting with GA fusion?
• RQ4 What does the attention map reveal about attribute/object grounding under various instructions?

Key findings

• GA units outperform concatenation units across multitask and zero-shot generalization in all difficulty modes.
• In hard mode, GA with A3C achieves 83% MT and 73% ZSL, vs 24% MT and 12% ZSL for Concat.
• GA models also outperform Concat under imitation learning (BC/DAgger), though exploration impacts IL in harder modes.
• Attention visualizations show dimension-specific gating corresponding to attributes like color and object type, indicating successful grounding of instructed attributes.
• A3C GA models learn faster and converge to higher accuracies than A3C Concat in the reported settings.