[Paper Review] FiLM: Visual Reasoning with a General Conditioning Layer

Ethan Perez, Florian Strub+3 more|arXiv (Cornell University)|Sep 22, 2017

Multimodal Machine Learning ApplicationsComputer Science27 references184 citations

TL;DR

FiLM introduces feature-wise linear modulation conditioned on text questions to guide CNNs, achieving state-of-the-art visual reasoning on CLEVR and strong generalization, including zero-shot scenarios.

ABSTRACT

We introduce a general-purpose conditioning method for neural networks called FiLM: Feature-wise Linear Modulation. FiLM layers influence neural network computation via a simple, feature-wise affine transformation based on conditioning information. We show that FiLM layers are highly effective for visual reasoning - answering image-related questions which require a multi-step, high-level process - a task which has proven difficult for standard deep learning methods that do not explicitly model reasoning. Specifically, we show on visual reasoning tasks that FiLM layers 1) halve state-of-the-art error for the CLEVR benchmark, 2) modulate features in a coherent manner, 3) are robust to ablations and architectural modifications, and 4) generalize well to challenging, new data from few examples or even zero-shot.

Motivation & Objective

• Motivate a general-purpose conditioning mechanism for neural networks for visual reasoning.
• Propose FiLM layers that apply a feature-wise affine transformation conditioned on input (e.g., a question).
• Demonstrate FiLM’s effectiveness across multiple visual reasoning tasks and analyze its learning dynamics and robustness.

Proposed method

• Define FiLM as FiLM(F_i,c | gamma_i,c, beta_i,c) = gamma_i,c * F_i,c + beta_i,c with gamma and beta produced from conditioning input via a FiLM generator.
• Use a question-processing GRU to produce FiLM parameters for each residual block in a CNN-based visual pipeline.
• Process image through FiLM-ed ResBlocks with 128 feature maps and 14x14 spatial resolution, optionally using pre-trained feature extractors.
• Concatenate 2 coordinate feature maps to aid spatial reasoning, and train end-to-end with Adam, no data augmentation.
• Demonstrate that FiLM can operate with both raw pixel inputs and learned image features, and analyze the role of gamma vs beta in conditioning.

Experimental results

Research questions

• RQ1 Can a general-purpose feature-wise affine conditioning mechanism enable effective multi-step visual reasoning?
• RQ2 How do FiLM parameters (gamma, beta) relate to different reasoning tasks (low-level vs high-level) across network layers?
• RQ3 What is FiLM’s robustness to architectural ablations and data regimes (few-shot, zero-shot)?
• RQ4 How well does FiLM generalize to human-posed questions and compositional generalization scenarios?
• RQ5 What is the impact of conditioning placement relative to normalization layers?

Key findings

• FiLM achieves state-of-the-art CLEVR accuracy, significantly reducing error compared to prior methods that do not use explicit reasoning.
• FiLM learns to selectively upregulate, downregulate, or shut off feature maps, enabling coherent and spatially localized reasoning.
• FiLM is robust to ablations and architectural changes and can learn effectively from limited data, generalizing to more complex or different data.
• FiLM generalizes well to CLEVR-Humans and, with fine-tuning, achieves substantial gains over prior methods on human-posed questions.
• A zero-shot generalization method in FiLM improves accuracy by about 3.2 percentage points on a subset of CoGenT-like tasks, indicating compositional control.
• The diversity of FiLM’s conditioning across layers suggests a function-based modularity emerging without explicit architectural priors.