[Paper Review] Learning Effective Visual Relationship Detector on 1 GPU

Yichao Lu, Cheng Chang+3 more|arXiv (Cornell University)|Dec 12, 2019

Advanced Neural Network Applications13 references4 citations

TL;DR

This paper presents a novel partial weight transfer method that enables training a complete visual relationship detection pipeline on a single GPU in under two days, achieving state-of-the-art performance on the Open Images 2019 Visual Relationship Challenge. By fine-tuning an object detector with class-matched pre-trained weights and combining spatio-semantic and visual features through a learnable fusion model, the approach achieves first place, outperforming the second-place team by over 5% on the private leaderboard.

ABSTRACT

We present our winning solution to the Open Images 2019 Visual Relationship challenge. This is the largest challenge of its kind to date with nearly 9 million training images. Challenge task consists of detecting objects and identifying relationships between them in complex scenes. Our solution has three stages, first object detection model is fine-tuned for the challenge classes using a novel weight transfer approach. Then, spatio-semantic and visual relationship models are trained on candidate object pairs. Finally, features and model predictions are combined to generate the final relationship prediction. Throughout the challenge we focused on minimizing the hardware requirements of our architecture. Specifically, our weight transfer approach enables much faster optimization, allowing the entire architecture to be trained on a single GPU in under two days. In addition to efficient optimization, our approach also achieves superior accuracy winning first place out of over 200 teams, and outperforming the second place team by over $5\%$ on the held-out private leaderboard.

Motivation & Objective

• To reduce the hardware requirements for training large-scale visual relationship detection models, making them accessible to researchers with limited resources.
• To improve training efficiency and convergence speed for visual relationship detection on massive datasets like Open Images V5.
• To achieve state-of-the-art performance on the Open Images 2019 Visual Relationship Challenge while minimizing computational cost.
• To develop a robust pipeline that combines object detection, spatio-semantic reasoning, and visual feature modeling for accurate relationship prediction.

Proposed method

• Proposes a partial weight transfer strategy that initializes classification and regression heads in a Faster R-CNN-based detector using pre-trained weights from COCO and Open Images, matching classes between source and target datasets.
• Fine-tunes the object detector on the challenge dataset using only minimal updates, significantly accelerating convergence compared to random initialization.
• Trains two separate models on candidate object pairs: a gradient-boosting model for spatio-semantic features (spatial and class-based context), and a CNN-based model for visual features (image patches around object pairs).
• Introduces a third-stage model that learns to optimally combine predictions from the spatio-semantic and visual models via weighted fusion, improving generalization over simple averaging.
• Treats 'is' relationships (object-attribute) separately, using a dedicated model trained on object-attribute pairs with class-specific head initialization.
• Employs a three-stage pipeline: (1) object detection, (2) feature extraction for candidate pairs, (3) end-to-end fusion of features and predictions for final relationship output.

Experimental results

Research questions

• RQ1 Can a visual relationship detection system be trained efficiently on a single GPU without sacrificing performance?
• RQ2 How effective is partial weight transfer—specifically, initializing detector heads with class-matched pre-trained weights—for accelerating convergence on a new, large-scale visual relationship dataset?
• RQ3 Can a learnable fusion mechanism outperform simple averaging of spatio-semantic and visual predictions in visual relationship detection?
• RQ4 Why do certain attributes (e.g., transparent, plastic) pose greater challenges for visual relationship detection, and how does training data distribution affect model performance?
• RQ5 To what extent can a multi-stage pipeline with separate modeling of spatial, semantic, and visual cues outperform end-to-end models in complex visual relationship detection?

Key findings

• The proposed partial weight transfer method reduced training time to under two days on a single GPU, enabling efficient training for researchers with limited hardware.
• The model achieved first place in the Open Images 2019 Visual Relationship Challenge, outperforming the second-place team by over 5% on the private leaderboard.
• The third-stage fusion model improved performance on all five evaluated relationships compared to individual models, with a 5.1-point gain in mAP_rel on the 'at' relationship (11% relative improvement).
• The spatio-semantic model outperformed on 'at' and 'hits' relationships, while the visual model excelled on 'plays' and 'inside_of', validating the need for both modality types.
• The model struggled most with attributes like 'transparent' and 'plastic' due to label ambiguity and low training instance counts, particularly for common furniture objects.
• Qualitative analysis showed robustness in complex scenes (e.g., multiple musicians), but failures occurred due to 3D-to-2D projection ambiguity and occlusion, suggesting potential for improvement with depth or multi-view inputs.