[Paper Review] Compositional Feature Augmentation for Unbiased Scene Graph Generation

Lin Li, Guikun Chen+4 more|arXiv (Cornell University)|Aug 13, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes Compositional Feature Augmentation (CFA), a novel, model-agnostic method to mitigate bias in Scene Graph Generation (SGG) by increasing feature diversity of tail predicates through augmenting both intrinsic and extrinsic features of relation triplets. CFA decomposes triplet features into intrinsic (object-level) and extrinsic (contextual) components, then applies feature replacement and mixup to enrich underrepresented tail categories, achieving state-of-the-art performance on VG and GQA with significant gains in mean recall across metrics.

ABSTRACT

Scene Graph Generation (SGG) aims to detect all the visual relation triplets $<$ exttt{sub}, exttt{pred}, exttt{obj}$>$ in a given image. With the emergence of various advanced techniques for better utilizing both the intrinsic and extrinsic information in each relation triplet, SGG has achieved great progress over the recent years. However, due to the ubiquitous long-tailed predicate distributions, today's SGG models are still easily biased to the head predicates. Currently, the most prevalent debiasing solutions for SGG are re-balancing methods, \eg, changing the distributions of original training samples. In this paper, we argue that all existing re-balancing strategies fail to increase the diversity of the relation triplet features of each predicate, which is critical for robust SGG. To this end, we propose a novel Compositional Feature Augmentation ( extbf{CFA}) strategy, which is the first unbiased SGG work to mitigate the bias issue from the perspective of increasing the diversity of triplet features. Specifically, we first decompose each relation triplet feature into two components: intrinsic feature and extrinsic feature, which correspond to the intrinsic characteristics and extrinsic contexts of a relation triplet, respectively. Then, we design two different feature augmentation modules to enrich the feature diversity of original relation triplets by replacing or mixing up either their intrinsic or extrinsic features from other samples. Due to its model-agnostic nature, CFA can be seamlessly incorporated into various SGG frameworks. Extensive ablations have shown that CFA achieves a new state-of-the-art performance on the trade-off between different metrics.

Motivation & Objective

• To address the persistent bias in SGG models toward head predicates due to long-tailed predicate distributions.
• To overcome the limitation of existing re-balancing methods, which fail to increase feature diversity of tail predicates despite adjusting sample frequencies or loss weights.
• To develop a model-agnostic augmentation strategy that enhances feature space diversity for underrepresented predicates.
• To improve decision boundary learning in long-tailed SGG by enriching the distribution of triplet features.
• To achieve better trade-offs between mR@K and R@K across head, body, and tail predicates.

Proposed method

• CFA decomposes each relation triplet feature into intrinsic (object-specific) and extrinsic (contextual) components.
• For intrinsic feature augmentation, it replaces the entity features of tail predicate triplets with semantically and contextually similar entities from other samples, using clustering based on pattern, context, and semantic similarity.
• For extrinsic feature augmentation, it applies mixup between foreground and background triplet features using a learnable parameter θ to control the mixing ratio.
• The method is model-agnostic and can be seamlessly integrated into existing SGG frameworks such as Motifs.
• Feature selection for replacement or mixing is guided by similarity metrics to ensure semantic and contextual plausibility.
• The approach increases feature diversity without altering training data frequency or loss weighting, directly targeting the root cause of bias: sparse and non-diverse feature distributions.

Experimental results

Research questions

• RQ1 Can increasing the diversity of relation triplet features, rather than just adjusting sample frequency or loss weights, lead to better generalization in SGG?
• RQ2 Does separating intrinsic and extrinsic features enable more effective and semantically meaningful data augmentation for tail predicates?
• RQ3 Can a model-agnostic augmentation strategy that mixes or replaces features from different samples improve performance across head, body, and tail predicates?
• RQ4 How do different similarity metrics (pattern, context, semantic) affect the quality and diversity of augmented features?
• RQ5 What hyperparameter settings (e.g., number of clusters K, mixup ratio θ) yield the best trade-off between mR@K and R@K in unbiased SGG?

Key findings

• CFA achieves a new state-of-the-art performance on the VG dataset under the PredCls setting, with a mean recall of 46.2, outperforming the baseline Motifs model.
• The ablation study shows that combining all CFA components (intrinsic replacement, foreground mixup, background mixup) yields the highest mean recall (46.2), indicating synergistic improvement.
• Using only intrinsic feature replacement improves mR@K by 2.8%–3.4% but slightly reduces R@K, indicating limited impact due to insufficient diversity.
• Extrinsic feature mixup (both foreground and background) significantly boosts mR@K (e.g., 17.8% to 30.3% on mR@100) while maintaining strong R@K, showing its effectiveness in enriching context features.
• The optimal number of clusters K=15 in intrinsic CFA provides the best balance of feature diversity and selection range, outperforming K=40 and K=150.
• The optimal mixup parameter θ=0.5 balances the influence of query and context features, with performance degrading at both extremes (θ=0.0 or θ=1.0).