[Paper Review] Modulating early visual processing by language

Harm de Vries, Florian Strub+4 more|arXiv (Cornell University)|Jul 2, 2017

Multimodal Machine Learning Applications28 references295 citations

TL;DR

The paper introduces Conditional Batch Normalization (CBN) to modulate an entire pre-trained ResNet with language, creating MODERN, which improves VQA performance by conditioning visual processing on linguistic input, including early stages.

ABSTRACT

It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, where visual and linguistic input are mostly processed independently before being fused into a single representation. In this paper, we deviate from this classic pipeline and propose to modulate the \emph{entire visual processing} by linguistic input. Specifically, we condition the batch normalization parameters of a pretrained residual network (ResNet) on a language embedding. This approach, which we call MOdulated RESnet (\MRN), significantly improves strong baselines on two visual question answering tasks. Our ablation study shows that modulating from the early stages of the visual processing is beneficial.

Motivation & Objective

• Motivate and test whether language can influence early visual processing rather than only high-level visual concepts.
• Propose a lightweight, scalable mechanism (CBN) to modulate convolutional feature maps using linguistic embeddings.
• Demonstrate improvements over strong baselines on VQA tasks by applying language conditioning to multiple stages of a pretrained CNN.

Proposed method

• Introduce Conditional Batch Normalization (CBN) that predicts changes to BN parameters from a language embedding.
• Freeze pretrained CNN weights and learn deltas (Delta beta, Delta gamma) via a small MLP conditioned on the question embedding.
• Apply CBN across all residual blocks in a ResNet to form the MODERN architecture.
• Evaluate MODERN on VQAv1 and GuessWhat?! with attention-based and baseline VQA models.
• Compare against strong baselines (Baseline, Ft Stage 4, Ft BN) and other fusion methods (MLB, MUTAN, MCB).
• Show that modulating early stages yields gains beyond fine-tuning BN or last blocks.

Experimental results

Research questions

• RQ1 Does conditioning the entire visual processing pipeline on language improve VQA performance compared to traditional two-stream pipelines?
• RQ2 Is modulation of early CNN layers by language more beneficial than conditioning only later layers or BN parameters?
• RQ3 How does MODERN compare with state-of-the-art fusion methods on VQA datasets?
• RQ4 What is the impact of applying CBN to different subsets of ResNet stages?
• RQ5 Can language-conditioned normalization improve performance in visually grounded tasks beyond VQA (e.g., GuessWhat?!

Key findings

• MODERN improves baseline VQA accuracy from 58.05% to 60.82% on 224x224 inputs.
• Fine-tuning only BN parameters yields a smaller improvement (58.98%), while fine-tuning the last stage alone is less effective (56.91%).
• Conditioning BN on language (MODERN) yields a significant gain over baselines and achieves competitive results with larger input resolutions.
• With 448x448 inputs, MODERN reaches 62.16% (MODERN) and 63.01% (MODERN + MLB), approaching or surpassing several strong baselines.
• On GuessWhat?! Oracle, MODERN reduces test error to 25.06% (from 29.92% with raw features), with larger gains when using spatial/category info.
• Ablation shows that modulating all stages yields best performance, with diminishing returns when restricting to later stages.