[Paper Review] Per-Pixel Classification is Not All You Need for Semantic Segmentation

Bowen Cheng, Alexander G. Schwing+1 more|arXiv (Cornell University)|Jul 13, 2021

Advanced Neural Network ApplicationsComputer Science50 references167 citations

TL;DR

This paper introduces MaskFormer, a mask-classification model that unifies semantic- and instance-level segmentation using a single architecture, loss, and training procedure, outperforming per-pixel baselines on datasets with large class vocabularies.

ABSTRACT

Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.

Motivation & Objective

• Motivate a unified segmentation paradigm that moves beyond per-pixel classification.
• Propose a simple mask classification model (MaskFormer) that can handle semantic and instance-level segmentation with the same training setup.
• Show that mask classification can outperform per-pixel baselines as the number of classes grows.
• Demonstrate state-of-the-art results on ADE20K (semantic) and COCO panoptic datasets (panoptic) with MaskFormer.
• Provide an efficient, flexible architecture that can leverage existing per-pixel classification backbones.

Proposed method

• Formulate semantic segmentation as mask classification by predicting a set of binary masks each associated with a class score.
• Use a Transformer decoder to generate N per-segment embeddings; a segmentation head produces class scores and an MLP generates N mask embeddings.
• Compute binary masks by dot-product between per-pixel embeddings and mask embeddings followed by sigmoid; merge or aggregate masks depending on the task.
• Train with a shared mask-classification loss combining a cross-entropy classification loss and a binary mask loss (focal + dice).
• Optionally use bipartite matching for predicting a variable number of masks and align predictions with ground-truth segments.
• Infer by assigning pixels to the best matching mask-class pair and post-process for semantic/panoptic outputs.

Experimental results

Research questions

• RQ1 Can a single mask-classification model replace per-pixel classification for semantic segmentation?
• RQ2 Does MaskFormer unify semantic- and instance-level (panoptic) segmentation with the same model, loss, and training pipeline?
• RQ3 How does mask classification perform compared to per-pixel baselines as the number of classes increases?
• RQ4 What is the impact of using bipartite matching versus fixed matching in MaskFormer?
• RQ5 What are the accuracy and efficiency trade-offs when applying MaskFormer to diverse datasets (ADE20K, COCO-Stuff, Cityscapes, etc.)?

Key findings

• MaskFormer outperforms per-pixel classification baselines on datasets with large vocabularies (e.g., ADE20K-Full with 847 classes).
• MaskFormer achieves state-of-the-art semantic segmentation on ADE20K (mIoU 55.6 with Swin-Transformer backbone).
• MaskFormer achieves state-of-the-art panoptic segmentation on COCO (52.7 PQ).
• MaskFormer matches or surpasses DETR-style mask approaches while using a simpler loss without heavy auxiliary losses.
• MaskFormer demonstrates better recognition quality (RQ) on large-class-vocabulary datasets, with some trade-offs in pixel-level segmentation quality (SQ).
• The method reduces parameters and FLOPs compared to some pixel-based baselines while improving accuracy, especially as class count grows.