[Paper Review] Long Tail Visual Relationship Recognition with Hubless Regularized Relmix

Sherif Abdelkarim, Aniket Agarwal+6 more|arXiv (Cornell University)|Mar 25, 2020

Multimodal Machine Learning Applications56 references4 citations

TL;DR

This paper introduces Long-Tail Visual Relationship Recognition (LTVRR) and proposes a novel visiolinguistic hubless (ViLHub) loss and relationship Mixup (RelMix) augmentation to improve recognition performance on rare visual relationship classes. The method achieves state-of-the-art results on two new long-tail benchmarks, VG8K-LT and GQA-LT, significantly boosting tail-class performance while maintaining head-class accuracy.

ABSTRACT

Several approaches have been proposed in recent literature to alleviate the long-tail problem, mostly in the object classification task. We propose to study the task of Long-Tail Visual Relationship Recognition (LTVRR), which aims at generalizing on the structured long-tail distribution of visual relationships (e.g., rabbit grazing on grass). In this setup, subject, relation, and object classes individually follow a long-tail distribution. We first introduce two large-scale long-tail visual relationship recognition benchmarks to study this task, dubbed as VG8K-LT (5330 objects, 2000 relationships) and GQA-LT (1703 objects, 310 relations). VG8K-LT and GQA-LT are built upon the widely used Visual Genome and GQA datasets. In contrast to existing benchmarks, some classes appear at a very low frequency ($1-14$ examples). We use these benchmarks to study the performance of several state-of-the-art long-tail models on LTVRR setup. We developed a visiolinguistic hubless (ViLHub) loss that consistently encourages visual classifiers to be more predictive of tail classes while being accurate on the head. We also propose relationship Mixup augmentation, dubbed as RelMix, to improve performance on the tail on VG8K-LT and GQA-LT benchmarks with the best performance achieved when combined with ViLHub loss. Benchmarks and code will be made available.

Motivation & Objective

• To address the long-tail distribution problem in visual relationship recognition, where subject, relation, and object classes are highly imbalanced.
• To develop new large-scale benchmarks, VG8K-LT and GQA-LT, that reflect realistic long-tail distributions with rare classes having only 1–14 examples.
• To propose a visiolinguistic hubless (ViLHub) loss that enhances prediction confidence for tail classes without sacrificing head-class accuracy.
• To design a relationship-specific Mixup augmentation (RelMix) that improves generalization on low-frequency visual relationships.
• To evaluate state-of-the-art long-tail methods on the new benchmarks and establish a new SOTA for LTVRR.

Proposed method

• The ViLHub loss is a visiolinguistic regularization technique that encourages visual classifiers to be more predictive of tail classes by minimizing distributional divergence between head and tail predictions.
• The loss is applied during training to align visual representations with linguistic priors, reducing overconfidence on head classes and improving generalization to rare relationships.
• RelMix performs interpolation between visual relationship samples using their subject-relation-object triplets, generating synthetic training examples that increase diversity and coverage of rare combinations.
• The method combines ViLHub loss with RelMix augmentation, enabling joint optimization for both head and tail class performance.
• The benchmarks VG8K-LT and GQA-LT are constructed by filtering Visual Genome and GQA datasets to retain only long-tail distributions with extreme class imbalance.
• Model training uses contrastive learning and multi-task supervision, with ViLHub and RelMix integrated into the loss and data augmentation stages respectively.

Experimental results

Research questions

• RQ1 How does the proposed ViLHub loss improve recognition performance on rare visual relationship classes compared to standard cross-entropy loss?
• RQ2 To what extent does the RelMix data augmentation strategy improve generalization on low-frequency visual relationships in long-tail settings?
• RQ3 How do existing state-of-the-art long-tail methods perform when evaluated on the newly introduced VG8K-LT and GQA-LT benchmarks?
• RQ4 What is the combined effect of ViLHub and RelMix on tail-class accuracy and overall model robustness?
• RQ5 Can the proposed method generalize across different long-tail visual relationship datasets with varying levels of class imbalance?

Key findings

• The ViLHub loss significantly improves prediction performance on tail classes, with measurable gains in F1-score for low-frequency relationships.
• RelMix augmentation alone boosts tail-class performance, and its combination with ViLHub yields the highest overall accuracy on both VG8K-LT and GQA-LT benchmarks.
• The proposed method achieves state-of-the-art performance on both VG8K-LT and GQA-LT, outperforming existing long-tail methods in both head and tail class recognition.
• The benchmarks VG8K-LT and GQA-LT reveal that existing models struggle with extreme long-tail distributions, especially for relationships with fewer than 10 training examples.
• The ablation study confirms that both ViLHub and RelMix contribute synergistically, with the best performance achieved when both components are used together.