[Paper Review] Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Ranjay Krishna, Yuke Zhu+10 more|arXiv (Cornell University)|Feb 23, 2016

Multimodal Machine Learning Applications251 citations

TL;DR

Visual Genome provides a densely annotated image dataset with objects, attributes, and relationships, plus region and scene graphs and region-based QA to enable cognitive scene understanding.

ABSTRACT

Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in an image. When asked "What vehicle is the person riding?", computers will need to identify the objects in an image as well as the relationships riding(man, carriage) and pulling(horse, carriage) in order to answer correctly that "the person is riding a horse-drawn carriage". In this paper, we present the Visual Genome dataset to enable the modeling of such relationships. We collect dense annotations of objects, attributes, and relationships within each image to learn these models. Specifically, our dataset contains over 100K images where each image has an average of 21 objects, 18 attributes, and 18 pairwise relationships between objects. We canonicalize the objects, attributes, relationships, and noun phrases in region descriptions and questions answer pairs to WordNet synsets. Together, these annotations represent the densest and largest dataset of image descriptions, objects, attributes, relationships, and question answers.

Motivation & Objective

• Motivate moving beyond object recognition to cognitive scene understanding by modeling interactions and relationships in images.
• Provide dense, region-level annotations (objects, attributes, relationships) canonicalized to WordNet synsets.
• Ground visual concepts to language and enable structured representations (region graphs and scene graphs).
• Offer region-based QA paired with descriptions to bridge vision and NLP tasks.
• Create a large, diverse dataset to benchmark models for comprehensive visual reasoning.

Proposed method

• Collect dense region descriptions for each image (average 42 per image) with bounding boxes.
• Extract and canonicalize objects, attributes, and relationships to WordNet synsets per region.
• Construct region graphs from region-level annotations and a unified scene graph for each image.
• Pair descriptions with region-based and freeform question-answer sets (region-based QA).
• Crowdsource data via Amazon Mechanical Turk with multiple task stages and quality controls.
• Provide seven data components: region descriptions, objects, attributes, relationships, region graphs, scene graphs, and QA pairs.

Experimental results

Research questions

• RQ1 How can we densely annotate images with objects, attributes, and relationships to enable cognitive understanding?
• RQ2 Can region graphs and scene graphs grounded to language improve visual reasoning tasks such as QA?
• RQ3 What is the impact of dense region-based descriptions and QA on grounding visual concepts to NLP representations?
• RQ4 How can WordNet synset canonicalization help cross-image querying and knowledge transfer?
• RQ5 What scale and diversity of annotations are needed to move toward comprehensive scene understanding?

Key findings

• The dataset covers 108,249 images with an average of 21 objects, 18 attributes, and 18 relationships per image.
• Each image includes about 42 region descriptions and 17 region-based QA pairs on average.
• Overall, 1.7 million QA pairs were collected, surpassing previous QA datasets in size.
• Objects, attributes, and relationships are canonicalized to WordNet synsets to enable standardized querying.
• Region graphs (per region) and a combined scene graph (per image) provide structured representations.
• Visual Genome emphasizes dense, region-based descriptions to capture multiple “stories” within an image.