[Paper Review] Imagination improves Multimodal Translation

Desmond Elliott, Ákos Kádár|arXiv (Cornell University)|May 11, 2017

Natural Language Processing Techniques46 references76 citations

TL;DR

The paper presents Imagination, a multitask model that learns translation and visually grounded representations by sharing an encoder, enabling state-of-the-art results on Multi30K without using images as input during translation. External data (described images and parallel text) can further boost performance.

ABSTRACT

We decompose multimodal translation into two sub-tasks: learning to translate and learning visually grounded representations. In a multitask learning framework, translations are learned in an attention-based encoder-decoder, and grounded representations are learned through image representation prediction. Our approach improves translation performance compared to the state of the art on the Multi30K dataset. Furthermore, it is equally effective if we train the image prediction task on the external MS COCO dataset, and we find improvements if we train the translation model on the external News Commentary parallel text.

Motivation & Objective

• Motivate and improve multimodal translation by leveraging visually grounded representations.
• Decompose multimodal translation into translation learning and visual grounding tasks.
• Show that shared encoders enable training with external parallel text or described images.
• Demonstrate state-of-the-art results on Multi30K without using images at translation time.
• Illustrate robustness to out-of-domain data and describe resources that aid performance.

Proposed method

• Propose a multitask model (Imagination) with a Shared Encoder feeding both a Translation Decoder and an Imaginet Image-Prediction Decoder.
• Train translation via an attention-based neural machine translation model.
• Train an auxiliary image-prediction decoder to predict a global image feature vector from the shared encoder representations.
• Use a joint objective J(θ,φ)=w J_T(θ,φ^t)+(1−w) J_G(θ,φ^g) to combine tasks and allow training on separate datasets (D_image, D_text).
• Evaluate with in-domain (Multi30K) and out-of-domain data (MS COCO, News Commentary) and with ensemble decoding.

Experimental results

Research questions

• RQ1 Can a shared encoder with multitask learning improve multimodal translation without using images at translation time?
• RQ2 Does auxiliary image prediction encourage grounded source representations that improve translation quality?
• RQ3 Is performance robust when training image grounding on external data and translation on external parallel text?
• RQ4 How do out-of-domain resources (COCO, News Commentary) affect translation quality when used in a multitask setting?
• RQ5 Does ensembling across domain-trained models yield state-of-the-art results on Multi30K?

Key findings

• The Imagination model achieves 55.8 Meteor and 36.8 BLEU (in-domain single model) and 57.6 Meteor (ensemble) on En→De Multi30K.
• With out-of-domain COCO-described images for image prediction, Imagination remains competitive (55.6 Meteor, 36.4 BLEU).
• Incorporating News Commentary parallel text yields improvements, reaching 59.3 Meteor in ensemble with COCO and Multi30K data.
• The model trained on in-domain data plus external resources achieves a new state-of-the-art of 59.3 Meteor in ensemble.
• Training with separate D_text and D_image data does not hurt performance; image prediction provides gains even when translation data is external.
• Using sub-word vocabularies and data combinations (Multi30K + NC) yields notable gains (e.g., 59.3 Meteor in the best ensemble).
• The Imaginet decoder’s grounding objective improves image retrieval alignment (median rank ~11) indicating grounded representations.