[Paper Review] Semantic Compositional Learning for Low-shot Scene Graph Generation

Tao He, Lianli Gao+3 more|arXiv (Cornell University)|Aug 19, 2021

Multimodal Machine Learning Applications31 references4 citations

TL;DR

This paper proposes a semantic compositional learning strategy to address low-shot and zero-shot scene graph generation by decomposing relation triples into essential and unessential components, then composing new, realistic triples using semantically or visually similar objects. The method significantly boosts performance across three SGG benchmarks, improving state-of-the-art models by nearly 50% on average in retrieval metrics.

ABSTRACT

Scene graphs provide valuable information to many downstream tasks. Many scene graph generation (SGG) models solely use the limited annotated relation triples for training, leading to their underperformance on low-shot (few and zero) scenarios, especially on the rare predicates. To address this problem, we propose a novel semantic compositional learning strategy that makes it possible to construct additional, realistic relation triples with objects from different images. Specifically, our strategy decomposes a relation triple by identifying and removing the unessential component and composes a new relation triple by fusing with a semantically or visually similar object from a visual components dictionary, whilst ensuring the realisticity of the newly composed triple. Notably, our strategy is generic and can be combined with existing SGG models to significantly improve their performance. We performed a comprehensive evaluation on the benchmark dataset Visual Genome. For three recent SGG models, adding our strategy improves their performance by close to 50\%, and all of them substantially exceed the current state-of-the-art.

Motivation & Objective

• Address the data scarcity problem in scene graph generation, especially for rare and unseen predicates in low-shot and zero-shot scenarios.
• Overcome the limitation of existing models that rely solely on annotated training triples and fail on rare relations.
• Enable effective learning for rare and unseen relation triples without requiring additional annotations.
• Develop a generic, model-agnostic strategy that can be integrated with existing SGG frameworks to improve performance.
• Improve zero-shot relation detection by composing novel, realistic relation triples using semantic and visual similarity.

Proposed method

• Decompose a relation triple $<$s, p, o$>$ into an essential component (e.g., subject and predicate) and an unessential component (e.g., object) based on the relative impact on the predicate.
• Construct a visual components dictionary containing diverse instances of object categories to serve as candidates for composition.
• Use a similarity measurement to select a semantically or visually similar object from the dictionary to replace the unessential component.
• Compose new relation triples by fusing the essential component with the selected object instance, ensuring realism through similarity-based selection.
• Integrate the compositional strategy as a data augmentation technique that operates independently of the main SGG model’s feature learning mechanism.
• Apply both intra-class (same category) and inter-class (semantically similar categories) composition to generate diverse, realistic training samples for rare and unseen relations.

Experimental results

Research questions

• RQ1 Can a model-agnostic strategy improve low-shot and zero-shot scene graph generation by generating realistic, additional training triples?
• RQ2 How effective is semantic and visual composition in alleviating data scarcity for rare predicates in scene graph generation?
• RQ3 To what extent can compositional learning enhance performance on few-shot and zero-shot relation retrieval tasks compared to existing methods?
• RQ4 Does the proposed similarity-based selection of replacement objects preserve realism and improve generalization in unseen relation scenarios?
• RQ5 Can the compositional strategy be seamlessly integrated with existing SGG models to yield consistent performance gains across different architectures?

Key findings

• The proposed DeC (Semantic Compositional Learning) strategy improves performance by nearly 50% on average across three SGG benchmarks (RR, FR, ZR) compared to current state-of-the-art models.
• On the rare predicates task (PredCls), the method achieves 35.7/38.6 mR@50/100 when applied to Motifs, significantly outperforming the previous best model TDE (13.2/17.1).
• In zero-shot relation retrieval (ZR), the method improves the best baseline (TDE) from 13.2/17.1 to 13.6/16.5 mR@50/100, demonstrating strong generalization to unseen relations.
• Ablation studies confirm that both intra-class and inter-class composition contribute positively, with intra-class being more effective for few-shot and intra-class for zero-shot settings.
• Replacing the similarity-based selection with random selection degrades performance across all tasks, proving the necessity of the proposed similarity measurement.
• The method achieves state-of-the-art performance on all three evaluation tasks—relation retrieval (RR), few-shot relation retrieval (FR), and zero-shot relation retrieval (ZR)—across three recent SGG models.