[Paper Review] Pixels to Graphs by Associative Embedding

Alejandro Newell, Jia Deng|arXiv (Cornell University)|Jun 22, 2017

Multimodal Machine Learning Applications23 references100 citations

TL;DR

The paper presents a single-stage end-to-end CNN that predicts a complete scene graph directly from pixels using associative embeddings, achieving state-of-the-art results on Visual Genome without relying on external proposal systems.

ABSTRACT

Graphs are a useful abstraction of image content. Not only can graphs represent details about individual objects in a scene but they can capture the interactions between pairs of objects. We present a method for training a convolutional neural network such that it takes in an input image and produces a full graph definition. This is done end-to-end in a single stage with the use of associative embeddings. The network learns to simultaneously identify all of the elements that make up a graph and piece them together. We benchmark on the Visual Genome dataset, and demonstrate state-of-the-art performance on the challenging task of scene graph generation.

Motivation & Objective

• Motivate end-to-end graph prediction from images to capture objects and their relations in a scene graph.
• Introduce associative embeddings as a means to connect graph vertices (objects) and edges (relations) without predefined slots.
• Develop a training scheme for unordered and overlapping detections grounded in pixel space.
• Benchmark on Visual Genome and demonstrate significant improvements over prior work.

Proposed method

• Ground graph elements (vertices and edges) at per-pixel locations using a high-resolution CNN output (heatmaps for vertices and edges).
• Predict vertex/edge properties (class, bounding box, relation type) from per-pixel features via fully connected heads.
• Use associative embeddings where each vertex gets a unique embedding; edges reference source/target embeddings to form the graph.
• Train with a pull-push embedding loss to keep embeddings of the same vertex together and different vertices apart, with a margin m=8 and embedding dimension d=8.
• Handle overlapping detections by using multiple slots per pixel and a Hungarian matching step during training to assign ground-truth elements to slots.
• Connect elements by matching edge embeddings to corresponding vertex embeddings to build the final graph.

Experimental results

Research questions

• RQ1 Can a CNN predict an entire scene graph (objects and their relationships) directly from an image without region proposals?
• RQ2 How can associative embeddings be used to reference and connect graph elements (vertices and edges) in an unordered output setting?
• RQ3 How to handle multiple graph elements grounded at the same pixel location during training and inference?
• RQ4 What performance gains can be achieved on Visual Genome for scene graph generation?

Key findings

• Achieves state-of-the-art results on Visual Genome scene graph generation across SGGen, SGCls, and PredCls tasks.
• Without RPNs, the model yields competitive Recall@K scores, demonstrating effective end-to-end graph prediction from raw images.
• Models with associative embeddings and unordered output slots significantly outperform prior approaches on the Visual Genome benchmarks.
• Longer training with efficient code yields updated gains (e.g., 03/2018 results show substantial improvements across settings).
• The approach reveals biases in predicate distributions and demonstrates how slots organize by predicate subsets (e.g., behind, has, in, of, on).