[Paper Review] ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph

F. Richard Yu, Jiji Tang+5 more|arXiv (Cornell University)|Jun 30, 2020

Multimodal Machine Learning ApplicationsComputer Science32 references118 citations

TL;DR

ERNIE-ViL introduces Scene Graph Prediction tasks that inject structured scene knowledge into vision-language pre-training, achieving state-of-the-art results on five downstream tasks and leading the VCR leaderboard by 3.7% absolute.

ABSTRACT

We propose a knowledge-enhanced approach, ERNIE-ViL, which incorporates structured knowledge obtained from scene graphs to learn joint representations of vision-language. ERNIE-ViL tries to build the detailed semantic connections (objects, attributes of objects and relationships between objects) across vision and language, which are essential to vision-language cross-modal tasks. Utilizing scene graphs of visual scenes, ERNIE-ViL constructs Scene Graph Prediction tasks, i.e., Object Prediction, Attribute Prediction and Relationship Prediction tasks in the pre-training phase. Specifically, these prediction tasks are implemented by predicting nodes of different types in the scene graph parsed from the sentence. Thus, ERNIE-ViL can learn the joint representations characterizing the alignments of the detailed semantics across vision and language. After pre-training on large scale image-text aligned datasets, we validate the effectiveness of ERNIE-ViL on 5 cross-modal downstream tasks. ERNIE-ViL achieves state-of-the-art performances on all these tasks and ranks the first place on the VCR leaderboard with an absolute improvement of 3.7%.

Motivation & Objective

• Motivate improving vision-language pre-training by capturing detailed cross-modal semantics (objects, attributes, relationships).
• Integrate structured knowledge from scene graphs into pre-training to strengthen cross-modal alignments.
• Demonstrate that scene-graph-guided pre-training yields gains across multiple cross-modal benchmarks.

Proposed method

• Construct Scene Graph Prediction tasks by masking and predicting objects, attributes, and relationships from parsed scene graphs of sentences.
• Use a two-stream cross-modal Transformer architecture to jointly model image regions and text with cross-modal attention.
• Pre-train with a combination of Scene Graph Prediction losses, MLM, Masked Region Prediction, and Image-Text Matching.
• Scene graphs parsed from text guide targeted predictions: Object Prediction, Attribute Prediction, Relationship Prediction.
• Object/Attribute/Relationship nodes are masked with specific strategies and recovered using context from both text and image regions.

Experimental results

Research questions

• RQ1 Can incorporating structured scene-graph knowledge during pre-training improve fine-grained vision-language understanding?
• RQ2 Do Scene Graph Prediction tasks lead to better cross-modal alignment for objects, attributes, and relationships across modalities?
• RQ3 How does ERNIE-ViL perform on standard vision-language benchmarks (VCR, VQA, RefCOCO+, Flickr-based retrieval) relative to prior pre-training methods?
• RQ4 What is the impact of in-domain vs out-of-domain pre-training data when using scene-graph-guided objectives?

Key findings

• Achieves state-of-the-art results on five downstream vision-language tasks.
• On VCR, ERNIE-ViL-large yields substantial gains over baselines, and the model ranks first on the VCR leaderboard with an absolute improvement of 3.7% on Q→AR over prior methods.
• Region-to-Phrase grounding (RefCOCO+) shows notable improvements (2.4% on test sets) when using Scene Graph Prediction.
• Pre-training with Scene Graph Prediction provides measurable gains across tasks and documentable improvements when comparing models initialized with ERNIE-2.0 or BERT.
• Cloze tests show the model with SGP tasks better predicts objects, attributes, and relationships, indicating stronger cross-modal detailed semantic understanding.