[Paper Review] Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction

Roei Herzig, Moshiko Raboh+3 more|arXiv (Cornell University)|Feb 15, 2018

Multimodal Machine Learning ApplicationsComputer Science28 references68 citations

TL;DR

The paper proposes graph-permutation invariant (GPI) deep architectures for structured prediction to generate scene graphs from images, proving a necessary-and-sufficient form and achieving state-of-the-art results on Visual Genome.

ABSTRACT

Machine understanding of complex images is a key goal of artificial intelligence. One challenge underlying this task is that visual scenes contain multiple inter-related objects, and that global context plays an important role in interpreting the scene. A natural modeling framework for capturing such effects is structured prediction, which optimizes over complex labels, while modeling within-label interactions. However, it is unclear what principles should guide the design of a structured prediction model that utilizes the power of deep learning components. Here we propose a design principle for such architectures that follows from a natural requirement of permutation invariance. We prove a necessary and sufficient characterization for architectures that follow this invariance, and discuss its implication on model design. Finally, we show that the resulting model achieves new state of the art results on the Visual Genome scene graph labeling benchmark, outperforming all recent approaches.

Motivation & Objective

• Motivate the use of permutation invariance for deep structured prediction in multi-object scenes.
• Derive a necessary and sufficient characterization of permutation-invariant architectures.
• Propose a GPI-based model for mapping images to scene graphs.
• Empirically demonstrate data-efficiency and state-of-the-art performance on Visual Genome.

Proposed method

• Define graph permutation invariance (GPI) for structured prediction outputs.
• Prove that a GPI function exists if and only if it can be implemented via a specific architecture: y_k = rho(z_k, sum_i alpha(z_i, sum_{j≠i} phi(z_i, z_{i,j}, z_j))) for all k.
• Show how phi, alpha, and rho enable aggregation of global graph information in a permutation-invariant way.
• Discuss extensions to incomplete graphs via automorphisms.
• Describe how attention and recurrent variants fit within the GPI framework.
• Present a Scene Graph Predictor (SGP) that uses GPI with entity and relation variables and attention over neighbors.

Experimental results

Research questions

• RQ1 How can permutation invariance guide the design of deep structured prediction architectures for graphs?
• RQ2 What is the exact architectural form that guarantees graph permutation invariance (GPI) and is also expressive enough for complex tasks like scene graph prediction?
• RQ3 Do GPI-based models improve data efficiency and performance on scene graph generation compared to non-GPI deep models?
• RQ4 Can attention and recurrent constructions be integrated within the GPI framework for scalable, parallelizable prediction?

Key findings

• GPI architectures require fewer samples to converge on correct solutions in synthetic graph labeling tasks.
• A GPI-based Scene Graph Predictor (SGP) with attention and linguistic features achieves state-of-the-art recall on SGCls and competitive results on PredCls on Visual Genome.
• Variants of GPI (no attention, neighbor attention, linguistic) consistently outperform baselines on constrained and unconstrained evaluations.
• The proposed model effectively learns to attend to relevant neighboring entities to refine predictions.