[Paper Review] Large-Scale Visual Relationship Understanding

Ji Zhang, Yannis Kalantidis+4 more|arXiv (Cornell University)|Apr 27, 2018

Multimodal Machine Learning Applications44 references15 citations

TL;DR

This paper proposes a novel deep learning model for large-scale visual relationship detection that jointly embeds visual and semantic features into a shared space, preserving both discriminative power and semantic similarity. It achieves state-of-the-art performance on benchmarks with over 53,000 object and 29,000 relation categories, even under extreme long-tailed class distributions, by using a multi-level feature fusion strategy and a contrastive triplet loss with semantic consistency regularization.

ABSTRACT

Large scale visual understanding is challenging, as it requires a model to handle the widely-spread and imbalanced distribution of triples. In real-world scenarios with large numbers of objects and relations, some are seen very commonly while others are barely seen. We develop a new relationship detection model that embeds objects and relations into two vector spaces where both discriminative capability and semantic affinity are preserved. We learn both a visual and a semantic module that map features from the two modalities into a shared space, where matched pairs of features have to discriminate against those unmatched, but also maintain close distances to semantically similar ones. Benefiting from that, our model can achieve superior performance even when the visual entity categories scale up to more than 80,000, with extremely skewed class distribution. We demonstrate the efficacy of our model on a large and imbalanced benchmark based of Visual Genome that comprises 53,000+ objects and 29,000+ relations, a scale at which no previous work has ever been evaluated at. We show superiority of our model over carefully designed baselines on the original Visual Genome dataset with 80,000+ categories. We also show state-of-the-art performance on the VRD dataset and the scene graph dataset which is a subset of Visual Genome with 200 categories.

Motivation & Objective

• To address the challenge of long-tailed, open-vocabulary visual relationship detection at scale, where relationships involve over 80,000 unique categories.
• To preserve both discriminative capability and semantic affinity in visual-semantic embedding spaces despite extreme class imbalance.
• To improve generalization on rare relationships by modeling semantic connections between subject, relation, and object through joint feature learning.
• To demonstrate effectiveness on large-scale, imbalanced benchmarks such as Visual Genome (VG80k) and standard datasets like VRD and scene graph.

Proposed method

• Introduces a dual-branch network that learns visual and semantic embeddings for subjects and objects separately, then fuses them at multiple levels before relation prediction.
• Employs a shared embedding space where matched visual-semantic pairs are pulled together, while unmatched pairs are pushed apart, using a contrastive triplet loss.
• Uses a novel consistency loss $\mathcal{L}_c$ to enforce that visual features with the same label (e.g., same object class) are embedded closely, preserving structural consistency.
• Applies a joint optimization of visual and semantic features using a multi-task loss combining contrastive, classification, and consistency terms.
• Utilizes late concatenation of subject and object features to provide strong contextual priors for relation prediction.
• Fine-tunes both visual and semantic representations jointly, rather than relying on pre-trained embeddings, to improve alignment and performance.

Experimental results

Research questions

• RQ1 Can a unified model effectively learn visual-semantic embeddings at scale, handling over 53,000 object and 29,000 relation categories?
• RQ2 How can a model maintain discriminative power while preserving semantic similarity in long-tailed, open-vocabulary visual relationship detection?
• RQ3 Does multi-level fusion of subject and object features improve relation prediction, especially for rare or complex relationships?
• RQ4 Can a contrastive loss with semantic consistency regularization outperform standard classification and triplet loss baselines in extreme long-tail scenarios?

Key findings

• The model achieves state-of-the-art performance on the Visual Genome dataset with over 80,000 categories, a scale previously untested by prior methods.
• On the VG80k validation set, the model achieves top-1 accuracy of 28.13% and mean recall of 47.48% with margin $m=0.2$, outperforming models without the consistency loss $\mathcal{L}_c$.
• The ablation study confirms that using all three losses ($\mathcal{L}_y$, $\mathcal{L}_x$, $\mathcal{L}_c$) yields the best performance, with $\mathcal{L}_c$ being critical for preserving structural consistency of visual features.
• The model generalizes well to rare relationships, producing semantically meaningful predictions even when incorrect, such as predicting "light on pole" instead of "lamp on pole".
• The triplet loss with $m=0.2$ achieves optimal performance, while larger margins degrade results due to treating all negatives equally, unlike the softmax-based contrastive approach used in this work.
• Qualitative results show the model successfully captures diverse relationship types—interactive, positional, attributive—across both frequent and rare categories.