[논문 리뷰] Modulating early visual processing by language

Harm de Vries, Florian Strub외 4명|arXiv (Cornell University)|2017. 07. 02.

Multimodal Machine Learning Applications참고 문헌 28인용 수 295

한 줄 요약

본 논문은 Conditional Batch Normalization (CBN)을 도입하여 언어로 전체 사전 학습된 ResNet을 모듈레이션하고, 이를 통해 MODERN을 생성하며, 이는 언어 입력에 따라 시각 처리의 초기 단계까지 포함하여 VQA 성능을 개선한다.

ABSTRACT

It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, where visual and linguistic input are mostly processed independently before being fused into a single representation. In this paper, we deviate from this classic pipeline and propose to modulate the \emph{entire visual processing} by linguistic input. Specifically, we condition the batch normalization parameters of a pretrained residual network (ResNet) on a language embedding. This approach, which we call MOdulated RESnet (\MRN), significantly improves strong baselines on two visual question answering tasks. Our ablation study shows that modulating from the early stages of the visual processing is beneficial.

연구 동기 및 목표

• Motivate and test whether language can influence early visual processing rather than only high-level visual concepts.
• Propose a lightweight, scalable mechanism (CBN) to modulate convolutional feature maps using linguistic embeddings.
• Demonstrate improvements over strong baselines on VQA tasks by applying language conditioning to multiple stages of a pretrained CNN.

제안 방법

• Introduce Conditional Batch Normalization (CBN) that predicts changes to BN parameters from a language embedding.
• Freeze pretrained CNN weights and learn deltas (Delta beta, Delta gamma) via a small MLP conditioned on the question embedding.
• Apply CBN across all residual blocks in a ResNet to form the MODERN architecture.
• Evaluate MODERN on VQAv1 and GuessWhat?! with attention-based and baseline VQA models.
• Compare against strong baselines (Baseline, Ft Stage 4, Ft BN) and other fusion methods (MLB, MUTAN, MCB).
• Show that modulating early stages yields gains beyond fine-tuning BN or last blocks.]
• research_questions:[

실험 결과

연구 질문

• RQ1 Does conditioning the entire visual processing pipeline on language improve VQA performance compared to traditional two-stream pipelines?
• RQ2 Is modulation of early CNN layers by language more beneficial than conditioning only later layers or BN parameters?
• RQ3 How does MODERN compare with state-of-the-art fusion methods on VQA datasets?
• RQ4 What is the impact of applying CBN to different subsets of ResNet stages?
• RQ5 Can language-conditioned normalization improve performance in visually grounded tasks beyond VQA (e.g., GuessWhat?!)

주요 결과

• MODERN improves baseline VQA accuracy from 58.05% to 60.82% on 224x224 inputs.
• Fine-tuning only BN parameters yields a smaller improvement (58.98%), while fine-tuning the last stage alone is less effective (56.91%).
• Conditioning BN on language (MODERN) yields a significant gain over baselines and achieves competitive results with larger input resolutions.
• With 448x448 inputs, MODERN reaches 62.16% (MODERN) and 63.01% (MODERN + MLB), approaching or surpassing several strong baselines.
• On GuessWhat?! Oracle, MODERN reduces test error to 25.06% (from 29.92% with raw features), with larger gains when using spatial/category info.
• Ablation shows that modulating all stages yields best performance, with diminishing returns when restricting to later stages.]
• table_headers: ["Image size", "Method", "Yes/No", "Number", "Other", "Overall"]
• table_rows: [["224x224", "Baseline", "79.45%", "36.63%", "44.62%", "58.05%"], ["224x224", "Ft Stage 4", "78.37%", "34.27%", "43.72%", "56.91%"], ["224x224", "Ft BN", "80.18%", "35.98%", "46.07%", "58.98%"], ["224x224", "MODERN", "81.17%", "37.79%", "48.66%", "60.82%"], ["448x448", "MLB [14] with ResNet-50", "80.20%", "37.73%", "49.53%", "60.84%"], ["448x448", "MLB [14] with ResNet-152", "80.95%", "38.39%", "50.59%", "61.73%"], ["448x448", "MUTAN + MLB [2]", "82.29%", "37.27%", "48.23%", "61.02%"], ["448x448", "MCB + Attention [9] with ResNet-50", "60.46%", "38.29%", "48.68%", "60.46%"], ["448x448", "MCB + Attention [9] with ResNet-152", "-", "-", "-", "62.50%"], ["448x448", "MODERN", "81.38%", "36.06%", "51.64%", "62.16%"], ["448x448", "MODERN + MLB [14]", "82.17%", "38.06%", "52.29%", "63.01%"]]} }# Answer to be corrected: The JSON structure above contains some misplaced content and an extra incomplete element after research_questions. Please provide the exact desired JSON structure, and I will finalize it correctly.
• title_en_US_placeholder