[Paper Review] Scene Graph Generation by Iterative Message Passing

Danfei Xu, Yuke Zhu+2 more|arXiv (Cornell University)|Jan 10, 2017

Multimodal Machine Learning ApplicationsComputer Science102 citations

TL;DR

The paper presents an end-to-end model that generates visually-grounded scene graphs from images by iteratively refining object and relation predictions through primal-dual message passing on a bipartite graph of nodes (objects) and edges (relationships).

ABSTRACT

Understanding a visual scene goes beyond recognizing individual objects in isolation. Relationships between objects also constitute rich semantic information about the scene. In this work, we explicitly model the objects and their relationships using scene graphs, a visually-grounded graphical structure of an image. We propose a novel end-to-end model that generates such structured scene representation from an input image. The model solves the scene graph inference problem using standard RNNs and learns to iteratively improves its predictions via message passing. Our joint inference model can take advantage of contextual cues to make better predictions on objects and their relationships. The experiments show that our model significantly outperforms previous methods for generating scene graphs using Visual Genome dataset and inferring support relations with NYU Depth v2 dataset.

Motivation & Objective

• Motivate the need for structured representations of visual scenes beyond object detection by explicitly modeling objects and their relationships as scene graphs.
• Propose an end-to-end neural model that jointly infers object categories, bounding boxes, and pairwise relationships from images.
• Leverage iterative message passing on a bipartite graph to incorporate contextual cues and refine predictions over iterations.
• Demonstrate the approach on a new Visual Genome-based scene graph dataset and on NYU Depth v2 for support relation reasoning.

Proposed method

• Extract object proposals with a Region Proposal Network (RPN) from the input image.
• Represent each object proposal with node GRUs and each object pair with edge GRUs sharing parameters across the graph.
• Use adaptive, learned message pooling to produce messages between nodes and edges and perform iterative GRU updates.
• Exploit a primal-dual bipartite graph structure to pass messages efficiently between node-GRUs and edge-GRUs in alternating iterations.
• Predict final object classes, bounding box refinements, and relationship predicates after several iterations.
• Train with cross-entropy for classification tasks and L1 loss for bounding box offsets; fine-tune only the fully connected layers and GRUs, with VGG-16 features frozen except FC layers.

Experimental results

Research questions

• RQ1 Can an end-to-end model generate visually-grounded scene graphs directly from images?
• RQ2 Does iterative message passing over a bipartite scene graph improve object and relationship predictions compared to per-element or non-iterative baselines?
• RQ3 How does the learned, adaptive message pooling affect information fusion across the graph?
• RQ4 To what extent can the model generalize to dense, spatial relationships like support relations in indoor scenes?

Key findings

• The final model with learned weighted message pooling outperforms a strong baseline across PredCls, SGCls, and SGGen tasks on Visual Genome (PredCls R@100: 53.08 vs 41.99; SGCls R@100: 24.38 vs 18.70; SGGen R@100: 4.24 vs 3.71).
• Two iterations of message passing yield the best predicate classification performance; additional iterations can introduce noisy messages and degrade results.
• The model achieves notable gains over the baseline both in frequent and less frequent predicates, thanks to contextual information and adaptive message aggregation (recall@5 for various predicates shows improved performance in our model).
• On NYU Depth v2, the approach achieves state-of-the-art results with RGB images for support relation prediction (type-aware: 89.0% vs 82.1% best prior; type-agnostic: 91.2% vs 88.4% best prior).
• The Visual Genome-based results include a refined dataset with average 25 objects and 22 relationships per image, enabling robust evaluation of scene graph generation tasks.