[Paper Review] Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity

Weiyao Wang, Matt Feiszli+3 more|arXiv (Cornell University)|Apr 12, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Generic Grouping Networks (GGNs) for open-world instance segmentation, leveraging a learned pairwise affinity predictor to generate pseudo-ground-truth masks from unlabeled images. By combining these pseudo-masks with real annotations, GGNs achieve state-of-the-art performance across COCO, LVIS, ADE20K, and UVO benchmarks, significantly outperforming closed-world models on unseen categories and demonstrating strong generalization to out-of-domain data.

ABSTRACT

Open-world instance segmentation is the task of grouping pixels into object instances without any pre-determined taxonomy. This is challenging, as state-of-the-art methods rely on explicit class semantics obtained from large labeled datasets, and out-of-domain evaluation performance drops significantly. Here we propose a novel approach for mask proposals, Generic Grouping Networks (GGNs), constructed without semantic supervision. Our approach combines a local measure of pixel affinity with instance-level mask supervision, producing a training regimen designed to make the model as generic as the data diversity allows. We introduce a method for predicting Pairwise Affinities (PA), a learned local relationship between pairs of pixels. PA generalizes very well to unseen categories. From PA we construct a large set of pseudo-ground-truth instance masks; combined with human-annotated instance masks we train GGNs and significantly outperform the SOTA on open-world instance segmentation on various benchmarks including COCO, LVIS, ADE20K, and UVO. Code is available on project website: https://sites.google.com/view/generic-grouping/.

Motivation & Objective

• To address the performance gap in instance segmentation for unseen categories in open-world settings, where existing models fail due to reliance on pre-defined class taxonomies.
• To develop a class-agnostic instance proposal method that generalizes across diverse, unseen object categories without requiring semantic supervision.
• To improve model generalization by leveraging pseudo-ground-truth masks derived from unsupervised, pairwise affinity learning on large-scale unlabeled data.
• To demonstrate that pre-training on pseudo-GT masks from unlabeled images yields stronger inductive bias than ImageNet supervision for open-world generalization.

Proposed method

• A Pairwise Affinity Predictor (PA) is trained to predict local pixel-level relationships using non-exhaustive segmentation masks as supervision, enabling generalization to unseen categories.
• The PA model generates affinity maps that are used as input to a grouping module to produce high-ranking, category-agnostic mask proposals, serving as pseudo-ground-truth masks.
• A class-agnostic Mask R-CNN is fine-tuned end-to-end using a combination of real human-annotated masks and the generated pseudo-GT masks to improve generic instance proposal capability.
• The method scales by pre-training on large-scale unlabeled datasets (e.g., ImageNet, OpenImages), where PA-based grouping generates pseudo-masks for additional data diversity.
• Performance is enhanced by combining multiple objectness scores: one from the PA-based grouping and another from the OLN (Open-World Learning Network) module.
• The approach is evaluated without fine-tuning on ADE20K and UVO, treating all masks as ground truth to simulate real-world open-world conditions.

Experimental results

Research questions

• RQ1 Can a model trained without semantic supervision achieve strong performance on open-world instance segmentation by leveraging unsupervised, local pixel relationships?
• RQ2 Does pseudo-ground-truth mask generation from pairwise affinity improve generalization to unseen categories compared to standard supervised pre-training?
• RQ3 How does pre-training on pseudo-GT masks from unlabeled images affect performance in open-world versus closed-world settings?
• RQ4 Can the combination of PA-based grouping and existing open-world models (e.g., OLN) further improve state-of-the-art performance?

Key findings

• GGNs significantly outperform Mask R-CNN and Selective Search on ADE20K and UVO, achieving 14 true positive proposals out of 20 ground truth objects, compared to 5 and 4 respectively.
• On the LVIS benchmark, GGN with pseudo-GT pre-training improves average recall (AR@100) by 6.8% over the previous SOTA (OLN), establishing a new SOTA.
• Pre-training on pseudo-GT masks from ImageNet improves open-world performance by 1.4% over ImageNet-supervised pre-training, despite underperforming slightly in closed-world settings.
• Performance improves with more pseudo-GT masks per image and more unlabeled pixels used during pre-training, indicating scalability and robustness to noise.
• The method generalizes well in real-world settings: when evaluated on ADE20K and UVO without fine-tuning, GGN achieves higher AR and AP than baselines on both seen and unseen categories.
• Combining objectness scores from PA-based grouping and OLN further improves performance, demonstrating complementary strengths in proposal ranking.