[Paper Review] Modulating and attending the source image during encoding improves Multimodal Translation

Jean-Benoit Delbrouck, Stéphane Dupont|arXiv (Cornell University)|Dec 9, 2017

Multimodal Machine Learning Applications17 references14 citations

TL;DR

This paper proposes a fully end-to-end multimodal translation model that modulates visual feature extraction using conditional batch normalization (CBN) and introduces an encoder-based visual attention mechanism conditioned on textual encoder representations. By integrating language-guided visual processing and attention during encoding, the model achieves new state-of-the-art results on three benchmark datasets, with BLEU scores up to 40.5 on Flickr Test2016 and 27.3 on the ambiguous MSCOCO set.

ABSTRACT

We propose a new and fully end-to-end approach for multimodal translation where the source text encoder modulates the entire visual input processing using conditional batch normalization, in order to compute the most informative image features for our task. Additionally, we propose a new attention mechanism derived from this original idea, where the attention model for the visual input is conditioned on the source text encoder representations. In the paper, we detail our models as well as the image analysis pipeline. Finally, we report experimental results. They are, as far as we know, the new state of the art on three different test sets.

Motivation & Objective

• Address the challenge of underperforming multimodal translation models by integrating visual and textual processing more deeply in the training pipeline.
• Overcome the limitations of conventional decoder-based attention on visual features by shifting attention computation to the encoder stage.
• Improve feature relevance for translation by modulating the convolutional neural network (CNN) processing using linguistic context from the source text encoder.
• Establish a fully end-to-end training paradigm where visual feature extraction is conditioned on textual input, rather than using fixed pre-extracted features.
• Demonstrate that language-conditioned visual processing leads to more informative image representations, especially in disambiguation tasks.

Proposed method

• Apply conditional batch normalization (CBN) to the ResNet encoder, where the batch normalization parameters are conditioned on the source text encoder’s hidden states to modulate feature extraction.
• Introduce an encoder-based visual attention mechanism that computes attention weights over visual features using the source word’s textual representation $\bm{h}_i$, rather than the decoder state.
• Use a bi-directional GRU encoder to generate contextualized textual annotations $\bm{h}_i$ for each source word, which are then used to condition both CBN and visual attention.
• Construct multimodal representations by fusing textual annotations $\bm{h}_i$ with corresponding visual features $V_i$ in the encoder, enabling joint processing before decoding.
• Train the entire model end-to-end using stochastic gradient descent to minimize negative log-likelihood of target sequences.
• Fine-tune the last ResNet stage or use global average pooling of the final feature map, comparing performance across different configurations.

Experimental results

Research questions

• RQ1 Can conditioning the visual feature extraction process on the source text encoder improve multimodal translation performance?
• RQ2 Does shifting visual attention computation from the decoder to the encoder stage lead to better alignment and translation quality?
• RQ3 How does conditional batch normalization (CBN) on the visual encoder affect the informativeness of image features in a multimodal setting?
• RQ4 What is the impact of using different ResNet variants (v1 vs. v2) on multimodal translation performance when combined with CBN and encoder-based attention?
• RQ5 Can a fully end-to-end multimodal translation model outperform models that use fixed, pre-extracted visual features?

Key findings

• The model with ResNet v1, CBN, and encoder-based attention (RN v1 CBN enc-att) achieved a BLEU score of 40.5 ± 0.8 on Flickr Test2016, surpassing the previous SOTA of 38.4 ± 0.3.
• On the ambiguous MSCOCO test set, the model reached a BLEU score of 27.3 ± 0.9 and METEOR of 48.5 ± 0.4, setting a new state-of-the-art.
• Conditional batch normalization (CBN) significantly improved performance, especially when applied to global image features (e.g., Pool5), with BLEU increasing from 38.4 to 39.4 on Test2016.
• Fine-tuning the last ResNet stage did not improve performance over fixed weights, suggesting that CBN alone is sufficient for effective modulation.
• The encoder-based attention mechanism outperformed decoder-based attention on visual features, indicating that early fusion of modality representations is more effective.
• ResNet v2, despite architectural improvements, led to a slight performance drop compared to ResNet v1, suggesting architectural choices may affect modality interaction in complex ways.