[Paper Review] 1st Place Solution for PSG competition with ECCV'22 SenseHuman Workshop

Qixun Wang, Xiaofeng Guo+1 more|arXiv (Cornell University)|Feb 6, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes GRNet, a two-stage panoptic scene graph generation framework that enhances relation prediction by integrating global context via a transformer-based global relation module, while addressing long-tailed predicate distribution and relation ambiguity through soft label supervision with self-distilled pseudo-labels. The method achieves state-of-the-art performance on the OpenPSG leaderboard, surpassing prior one-stage and two-stage methods.

ABSTRACT

Panoptic Scene Graph (PSG) generation aims to generate scene graph representations based on panoptic segmentation instead of rigid bounding boxes. Existing PSG methods utilize one-stage paradigm which simultaneously generates scene graphs and predicts semantic segmentation masks or two-stage paradigm that first adopt an off-the-shelf panoptic segmentor, then pairwise relationship prediction between these predicted objects. One-stage approach despite having a simplified training paradigm, its segmentation results are usually under-satisfactory, while two-stage approach lacks global context and leads to low performance on relation prediction. To bridge this gap, in this paper, we propose GRNet, a Global Relation Network in two-stage paradigm, where the pre-extracted local object features and their corresponding masks are fed into a transformer with class embeddings. To handle relation ambiguity and predicate classification bias caused by long-tailed distribution, we formulate relation prediction in the second stage as a multi-class classification task with soft label. We conduct comprehensive experiments on OpenPSG dataset and achieve the state-of-art performance on the leadboard. We also show the effectiveness of our soft label strategy for long-tailed classes in ablation studies. Our code has been released in https://github.com/wangqixun/mfpsg.

Motivation & Objective

• To address the performance gap between one-stage and two-stage paradigms in panoptic scene graph generation by enhancing two-stage methods with global context.
• To mitigate relation ambiguity and long-tailed distribution bias in predicate classification through a novel soft label training strategy.
• To improve scene graph generation performance by jointly optimizing all pairwise relations using a transformer-based global relation module.
• To achieve state-of-the-art results on the OpenPSG benchmark under standard evaluation protocols.

Proposed method

• GRNet employs a two-stage paradigm: first using Mask2Former as an off-the-shelf panoptic segmentor to extract object masks and RoI features.
• Object-level features and their corresponding masks are fused and fed into a transformer encoder with class embeddings to model global relations among all detected objects.
• Relation prediction is reformulated as a multi-class classification task using soft labels generated via exponential moving average (EMA) of model predictions during training.
• Self-distilled soft labels are used to reduce bias from long-tailed predicate distributions and handle missing annotations.
• Focal loss is applied to re-balance the training loss across rare and frequent predicate categories.
• The full model is fine-tuned end-to-end after pre-training the segmentor for 50 epochs, with joint training for 12 additional epochs.

Figure 1: Panoptic Scene graph Generation . PSG presents a more comprehensive and clean scene graph representation, with more accurate localization of objects and including relationships with the background (known as stuff).

Experimental results

Research questions

• RQ1 Can a two-stage panoptic scene graph generation method effectively recover global context to match the performance of one-stage methods?
• RQ2 How does soft label supervision with self-distilled pseudo-labels improve relation prediction in the presence of long-tailed predicate distributions?
• RQ3 To what extent does the global relation module enhance performance on distant object relations compared to local feature-based methods?
• RQ4 Does the combination of soft labels and focal loss lead to consistent improvements across both frequent and rare predicate classes?

Key findings

• GRNet with Swin-B backbone and Mask2Former segmentor achieves 55.0% mR@100 on the OpenPSG test set, surpassing all prior one-stage and two-stage methods.
• The method improves mR@100 by 9.1 percentage points over the previous best two-stage baseline (GPSNet) and 14.9 points over the best one-stage method (PSGFormer).
• Ablation studies confirm that soft label training significantly improves performance on long-tailed predicate classes, especially those with fewer than 100 training samples.
• The use of focal loss further enhances performance on rare predicates, reducing the performance gap between frequent and rare categories.
• The global relation module leads to a 7.5% absolute gain in mR@100 compared to a local feature-based baseline, demonstrating the value of global context in relation reasoning.
• The model achieves 40.8% mR@100 with Swin-B backbone, indicating strong scalability with deeper architectures.

Figure 2: The pipeline of our proposed GRNet . We follow two-stage paradigm where an off-theshelf panoptic segmentor is used to generate object mask and then pairwise relationship prediction between these predicted objects.