[Paper Review] Relationformer: A Unified Framework for Image-to-Graph Generation

Suprosanna Shit, Rajat Koner+9 more|Zurich Open Repository and Archive (University of Zurich)|Mar 19, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

Relationformer proposes a unified one-stage transformer-based framework for end-to-end image-to-graph generation, introducing a learnable [rln]-token to jointly model object-relation interactions. By integrating [obj]-tokens and [rln]-tokens for simultaneous object detection and relation prediction, it achieves state-of-the-art performance across diverse datasets including road networks, blood vessels, and scene graphs, with improved accuracy, speed, and parameter efficiency compared to two-stage baselines.

ABSTRACT

A comprehensive representation of an image requires understanding objects and their mutual relationship, especially in image-to-graph generation, e.g., road network extraction, blood-vessel network extraction, or scene graph generation. Traditionally, image-to-graph generation is addressed with a two-stage approach consisting of object detection followed by a separate relation prediction, which prevents simultaneous object-relation interaction. This work proposes a unified one-stage transformer-based framework, namely Relationformer, that jointly predicts objects and their relations. We leverage direct set-based object prediction and incorporate the interaction among the objects to learn an object-relation representation jointly. In addition to existing [obj]-tokens, we propose a novel learnable token, namely [rln]-token. Together with [obj]-tokens, [rln]-token exploits local and global semantic reasoning in an image through a series of mutual associations. In combination with the pair-wise [obj]-token, the [rln]-token contributes to a computationally efficient relation prediction. We achieve state-of-the-art performance on multiple, diverse and multi-domain datasets that demonstrate our approach's effectiveness and generalizability.

Motivation & Objective

• To address the limitations of two-stage image-to-graph generation pipelines that decouple object detection and relation prediction, leading to error propagation and inefficiency.
• To enable joint, end-to-end learning of objects and their relationships in a single model, leveraging mutual context and co-occurrence for improved reasoning.
• To introduce a novel learnable [rln]-token that captures interdependencies among objects and enhances global semantic awareness for relation prediction.
• To simplify the training and inference pipeline by eliminating hand-designed features and multi-stage modules, improving efficiency and scalability.
• To demonstrate generalizability and effectiveness across diverse domains, including 2D/3D structural graphs and semantic scene graphs.

Proposed method

• Proposes a unified one-stage transformer architecture based on DETR, using learnable [obj]-tokens for object detection and a novel [rln]-token for relation modeling.
• Introduces the [rln]-token as a learnable query that attends to object pairs and captures global semantic and spatial relationships through multi-head self-attention.
• Uses [obj]-[rln]-[obj] triplets to predict pairwise relations, enabling joint representation learning between objects and their relations.
• Employs a set-based prediction head with bipartite matching to predict object and relation outputs end-to-end without non-maximum suppression.
• Leverages multi-head cross-attention between [rln]-token and [obj]-tokens to allow all objects to attend to relation-aware features.
• Trains the model end-to-end using a query-agnostic detection head and a learnable [rln]-token that adapts to the input image context.

Experimental results

Research questions

• RQ1 Can a unified one-stage transformer framework jointly predict objects and their relations in image-to-graph generation, outperforming two-stage pipelines?
• RQ2 How does the introduction of a learnable [rln]-token improve the modeling of object co-occurrence and mutual dependencies?
• RQ3 What is the impact of the [rln]-token on relation prediction performance across diverse image-to-graph tasks?
• RQ4 Does the [rln]-token enable better generalization and efficiency compared to traditional two-stage or feature-based approaches?
• RQ5 How does training data size affect the performance of the [rln]-token in end-to-end graph generation?

Key findings

• Relationformer achieves state-of-the-art performance on road network, blood vessel, and scene graph generation benchmarks, outperforming existing two-stage methods.
• On the Visual Genome dataset, Relationformer with [rln]-token and MLP head achieves 31.3% mAP@100, a 2.9% improvement over the baseline without [rln]-token.
• In 3D synthetic vessel graph generation, Relationformer with [rln]-token and 100% training data achieves 84.3% mAR for edge detection, a 1.8% improvement over the def-DETR baseline.
• Ablation shows that the [rln]-token significantly improves relation prediction, especially when combined with an MLP head, indicating its role in modeling complex semantic relationships.
• The model demonstrates strong generalization, with performance improving consistently as training data size increases, suggesting scalability with more data.
• The framework reduces model complexity and inference time compared to two-stage pipelines, while using fewer parameters and less energy.