[Paper Review] Mask R-CNN

Kaiming He, Georgia Gkioxari+2 more|arXiv (Cornell University)|Mar 20, 2017

Advanced Neural Network Applications400 citations

TL;DR

Mask R-CNN extends Faster R-CNN with a parallel mask prediction branch, achieving state-of-the-art single-model instance segmentation on COCO and strong results in detection and keypoint tasks, aided by RoIAlign and per-class mask decoupling.

ABSTRACT

We present a conceptually simple, flexible, and general framework for object instance segmentation. Our approach efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework. We show top results in all three tracks of the COCO suite of challenges, including instance segmentation, bounding-box object detection, and person keypoint detection. Without bells and whistles, Mask R-CNN outperforms all existing, single-model entries on every task, including the COCO 2016 challenge winners. We hope our simple and effective approach will serve as a solid baseline and help ease future research in instance-level recognition. Code has been made available at: https://github.com/facebookresearch/Detectron

Motivation & Objective

• Develop a unified, simple framework for simultaneous object detection and instance segmentation.
• Improve pixel-to-pixel alignment for RoIs to enable accurate mask predictions.
• Decouple mask prediction from classification to allow per-class mask outputs.
• Evaluate Mask R-CNN across COCO tasks: instance segmentation, bounding-box detection, and keypoint detection.
• Demonstrate training/inference efficiency and framework versatility across backbone architectures.

Proposed method

• Extend Faster R-CNN with a parallel mask prediction branch for each RoI.
• Introduce RoIAlign to remove RoI misalignment caused by RoIPool via bilinear interpolation without quantization.
• Predict K binary masks per RoI, one per class, using a per-pixel sigmoid with a binary cross-entropy loss for the mask branch (L_mask).
• Use an FCN-based mask head to preserve spatial layout (m x m masks) instead of collapsing to vectors.
• Train with multi-task loss L = L_cls + L_box + L_mask, sharing features where possible.
• Instantiate Mask R-CNN with backbones like ResNet-50/101, FPN, and (optionally) ResNeXt, and evaluate on COCO.

Experimental results

Research questions

• RQ1 Does adding a parallel mask branch to Faster R-CNN yield accurate instance segmentation without sacrificing detection performance?
• RQ2 How does RoIAlign improve mask quality compared to RoIPool in the context of instance segmentation?
• RQ3 Is decoupling mask prediction from class prediction (per-class binary masks) beneficial for instance segmentation accuracy?
• RQ4 How do different backbones (ResNet, FPN, ResNeXt) affect mask, box, and keypoint performance within Mask R-CNN?
• RQ5 Can Mask R-CNN be effectively extended to other tasks such as human pose estimation while maintaining speed?

Key findings

• Mask R-CNN with ResNet-101-FPN achieves 35.7 mask AP on COCO test-dev, outperforming previous single-model methods.
• RoIAlign improves mask AP by about 3 points (and AP 75 by about 5 points) over RoIPool, with large gains at higher IoU.
• Decoupling mask and class prediction by using independent class-specific masks (binary sigmoid) yields +5.5 AP over a multinomial softmax approach.
• Backbone choices significantly affect performance; deeper networks plus FPN and ResNeXt yield notable gains (e.g., ResNeXt-101-FPN improves mask AP to 37.1).
• Mask R-CNN also improves bounding-box detection when used with the same backbone, and the gap between box and mask AP is reduced (e.g., 2.7 points in some configurations).
• Inference with feature sharing (RPN and Mask R-CNN) runs around 195 ms per image on a Tesla M40, and training on COCO takes 32–44 hours on 8 GPUs; multi-task training provides strong gains across tasks.
• The framework generalizes to human pose estimation by treating keypoints as one-hot masks, achieving competitive AP on COCO keypoint tasks.