[Paper Review] The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation

Beomyoung Kim, Joonhyun Jeong+2 more|arXiv (Cornell University)|Mar 27, 2023

Image and Object Detection Techniques4 citations

TL;DR

This paper proposes a weakly semi-supervised instance segmentation framework using point labels to improve pseudo-mask quality and reduce false positives/negatives in proposals. By leveraging point labels for adaptive pyramid-level selection and introducing a MaskRefineNet that fuses image, rough mask, and point inputs, the method achieves 38.8% AP on COCO with only 50% of the fully labeled data—nearly matching fully supervised performance (39.7%).

ABSTRACT

In this paper, we introduce a novel learning scheme named weakly semi-supervised instance segmentation (WSSIS) with point labels for budget-efficient and high-performance instance segmentation. Namely, we consider a dataset setting consisting of a few fully-labeled images and a lot of point-labeled images. Motivated by the main challenge of semi-supervised approaches mainly derives from the trade-off between false-negative and false-positive instance proposals, we propose a method for WSSIS that can effectively leverage the budget-friendly point labels as a powerful weak supervision source to resolve the challenge. Furthermore, to deal with the hard case where the amount of fully-labeled data is extremely limited, we propose a MaskRefineNet that refines noise in rough masks. We conduct extensive experiments on COCO and BDD100K datasets, and the proposed method achieves promising results comparable to those of the fully-supervised model, even with 50% of the fully labeled COCO data (38.8% vs. 39.7%). Moreover, when using as little as 5% of fully labeled COCO data, our method shows significantly superior performance over the state-of-the-art semi-supervised learning method (33.7% vs. 24.9%). The code is available at https://github.com/clovaai/PointWSSIS.

Motivation & Objective

• Address the performance gap in semi-supervised instance segmentation by reducing false-negative and false-positive proposals.
• Leverage low-cost point labels as weak supervision to guide proposal filtering and mask refinement.
• Overcome limitations in low-data regimes where mask predictions are noisy due to insufficient supervision.
• Develop a method that effectively balances proposal quality and mask accuracy using minimal fully labeled data and abundant point-labeled data.
• Enable high-performance instance segmentation under realistic budget constraints by combining point supervision with refined mask prediction.

Proposed method

• Use point labels to filter instance proposals, retaining only true-positive proposals to reduce false positives.
• Implement Adaptive Pyramid-Level Selection to assign each point to the most confident FPN feature level based on network confidence scores.
• Design MaskRefineNet that takes image, rough mask, and point as inputs to refine noisy mask predictions using attention-based fusion.
• Train a base network on fully labeled data, then generate pseudo masks using filtered proposals and refined masks for target network training.
• Apply a two-stage training pipeline: first train base model on fully labeled data, then use point-guided pseudo labels to train the target model.
• Use min-max point extraction from predicted masks to generate bounding boxes for evaluation on object detection benchmarks.

Experimental results

Research questions

• RQ1 Can point labels effectively reduce false-positive and false-negative proposals in semi-supervised instance segmentation?
• RQ2 How can point labels be used to improve mask quality when fully labeled data is scarce?
• RQ3 Does adaptive selection of FPN levels based on confidence improve mask prediction accuracy?
• RQ4 To what extent can a MaskRefineNet that fuses image, rough mask, and point inputs reduce noise in pseudo masks?
• RQ5 Can the proposed method achieve performance close to fully supervised learning with significantly less fully labeled data?

Key findings

• The proposed method achieves 38.8% AP on COCO using only 50% of the fully labeled data, compared to 39.7% for fully supervised learning.
• With just 5% of the fully labeled data, the method reaches 33.7% AP, significantly outperforming the state-of-the-art semi-supervised method (24.9% AP).
• On the COCO test-dev object detection benchmark, the method surpasses existing weakly semi-supervised object detection methods using the same ResNet-50 backbone.
• Qualitative results show that MaskRefineNet effectively separates overlapping instances and removes noisy pixels when all three inputs (image, rough mask, point) are used.
• On BDD100K, models trained with point-labeled data show improved robustness to small, large, and occluded objects compared to models without point supervision.
• Ablation studies confirm that all three inputs to MaskRefineNet are essential: omitting the point or rough mask leads to misclassification and inaccurate mask boundaries.