[Paper Review] SPAN: Learning Similarity between Scene Graphs and Images with Transformers

Yuren Cong, Wentong Liao+2 more|arXiv (Cornell University)|Apr 2, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes GICON, a contrastive learning framework using graph and image Transformers to measure semantic similarity between scene graphs and images. By serializing scene graphs with structural encoding, the model enables robust, graph-aware image retrieval and introduces R-Precision as a new, noise-resistant evaluation metric for scene graph generation, outperforming triplet-based metrics on Visual Genome and Open Images datasets.

ABSTRACT

Learning similarity between scene graphs and images aims to estimate a similarity score given a scene graph and an image. There is currently no research dedicated to this task, although it is critical for scene graph generation and downstream applications. Scene graph generation is conventionally evaluated by Recall$@K$ and mean Recall$@K$, which measure the ratio of predicted triplets that appear in the human-labeled triplet set. However, such triplet-oriented metrics fail to demonstrate the overall semantic difference between a scene graph and an image and are sensitive to annotation bias and noise. Using generated scene graphs in the downstream applications is therefore limited. To address this issue, for the first time, we propose a Scene graPh-imAge coNtrastive learning framework, SPAN, that can measure the similarity between scene graphs and images. Our novel framework consists of a graph Transformer and an image Transformer to align scene graphs and their corresponding images in the shared latent space. We introduce a novel graph serialization technique that transforms a scene graph into a sequence with structural encodings. Based on our framework, we propose R-Precision measuring image retrieval accuracy as a new evaluation metric for scene graph generation. We establish new benchmarks on the Visual Genome and Open Images datasets. Extensive experiments are conducted to verify the effectiveness of SPAN, which shows great potential as a scene graph encoder.

Motivation & Objective

• To address the limitations of triplet-oriented metrics like Recall@K, which fail to capture global semantic alignment between scene graphs and images.
• To develop a unified, contrastive learning framework that aligns scene graphs and images in a shared latent space using Transformers.
• To introduce a novel evaluation metric, R-Precision, based on image retrieval accuracy using generated scene graphs as queries.
• To establish new benchmarks on Visual Genome and Open Images using the proposed similarity-based evaluation.
• To demonstrate the effectiveness of a graph serialization technique that preserves structural information for graph Transformer encoders.

Proposed method

• A graph serialization technique transforms scene graphs into sequences with structural encoding, enabling the graph Transformer to process graph-structured data.
• Two separate Transformers (graph and image) are trained via contrastive learning to align scene graphs and their corresponding images in a shared latent space.
• The framework uses a contrastive loss to maximize similarity between positive image-scene graph pairs and minimize similarity for negative pairs.
• R-Precision is computed by retrieving the top-K most similar images from a candidate set and measuring the proportion of correctly retrieved ground-truth images.
• The method supports both location-free and location-bound scene graphs, with bounding box information integrated into the graph representation.
• Ablation studies evaluate the impact of Transformer depth and architecture on performance, with 6 layers found optimal.

Experimental results

Research questions

• RQ1 Can a contrastive learning framework based on Transformers effectively measure semantic similarity between scene graphs and images?
• RQ2 Does the proposed graph serialization technique preserve structural information well enough to enable effective graph representation learning?
• RQ3 How does R-Precision compare to Recall@K in evaluating scene graph generation quality, especially under data bias or noise?
• RQ4 Can the proposed framework improve image retrieval performance when using generated scene graphs as queries?
• RQ5 What is the optimal depth of the graph and image Transformers for balancing performance and overfitting?

Key findings

• The proposed GICON framework achieves an R-Precision of 78.9% on the Open Images V6 dataset using location-bound scene graphs with K=100.
• On the Visual Genome dataset, the model reaches 76.1% R-Precision with location-bound graphs and K=100, demonstrating strong retrieval performance.
• The optimal model configuration uses 6 layers in both the graph and image Transformers, as deeper networks (8 layers) lead to performance degradation due to overfitting.
• The framework successfully retrieves the correct image as the top result in 78.9% of cases on Open Images, outperforming CLIP when using scene graph queries.
• Qualitative results show that the model correctly identifies semantically similar images even when triplet-level metrics fail due to label bias.
• The average similarity between ground-truth scene graphs and images is 0.56 on both Visual Genome and Open Images, indicating consistent alignment in the latent space.