[Paper Review] SOLOv2: Dynamic and Fast Instance Segmentation

Xinlong Wang, Rufeng Zhang+3 more|arXiv (Cornell University)|Mar 23, 2020

Advanced Neural Network ApplicationsComputer Science40 references476 citations

TL;DR

SOLOv2 introduces a box-free, fully convolutional instance segmentation framework that predicts instance masks by location using dynamically generated kernels and a unified high-resolution mask feature, paired with a fast Matrix NMS for post-processing; it achieves state-of-the-art speed/accuracy on COCO and LVIS.

ABSTRACT

In this work, we aim at building a simple, direct, and fast instance segmentation framework with strong performance. We follow the principle of the SOLO method of Wang et al. "SOLO: segmenting objects by locations". Importantly, we take one step further by dynamically learning the mask head of the object segmenter such that the mask head is conditioned on the location. Specifically, the mask branch is decoupled into a mask kernel branch and mask feature branch, which are responsible for learning the convolution kernel and the convolved features respectively. Moreover, we propose Matrix NMS (non maximum suppression) to significantly reduce the inference time overhead due to NMS of masks. Our Matrix NMS performs NMS with parallel matrix operations in one shot, and yields better results. We demonstrate a simple direct instance segmentation system, outperforming a few state-of-the-art methods in both speed and accuracy. A light-weight version of SOLOv2 executes at 31.3 FPS and yields 37.1% AP. Moreover, our state-of-the-art results in object detection (from our mask byproduct) and panoptic segmentation show the potential to serve as a new strong baseline for many instance-level recognition tasks besides instance segmentation. Code is available at: https://git.io/AdelaiDet

Motivation & Objective

• Motivate a simpler, box-free approach to instance segmentation that avoids bounding box detectors.
• Develop a dynamic, location-conditioned mask generation mechanism to produce high-resolution instance masks.
• Eliminate bottlenecks in mask prediction and post-processing to improve speed without sacrificing accuracy.
• Demonstrate strong performance across COCO and LVIS, including extensions to object detection and panoptic segmentation.

Proposed method

• Predict mask kernels dynamically for each location using a learned D-dimensional kernel conditioned on image features.
• Compute a unified, high-resolution mask feature representation that is shared across FPN levels.
• Convolve the dynamically generated kernels with the mask features to produce per-location instance masks.
• Use CoordConv-enhanced inputs to inject explicit spatial coordinates into the mask kernel branch.
• Apply Matrix NMS to suppress duplicate mask predictions in parallel, improving speed and accuracy.
• Optionally derive bounding boxes from predicted masks to obtain box-based results without separate box training.

Experimental results

Research questions

• RQ1 Can a direct, box-free framework achieve competitive instance segmentation by predicting masks conditioned on location?
• RQ2 Does separating mask kernel learning from mask feature learning improve efficiency and accuracy?
• RQ3 Can a parallel, matrix-based NMS (Matrix NMS) outperform traditional NMS and Fast NMS for masks?
• RQ4 What is the impact of explicit coordinate information and unified mask features on mask quality across object scales?
• RQ5 How does SOLOv2 perform on COCO and LVIS in both accuracy and speed, and can it extend to detection and panoptic segmentation?

Key findings

• SOLOv2 achieves state-of-the-art speed-accuracy trade-offs on COCO, e.g., 38.8% AP with ResNet-50-FPN at 18 FPS and 37.1% AP in a light-weight version.
• SOLOv2 with Res-DCN-101-FPN reaches 41.7% mask AP and 61.6 mAP for box-based detection under COCO test-dev setup.
• Matrix NMS processes 500 masks in under 1 ms and outperforms Fast NMS by 0.4% AP.
• Unified mask feature representation yields better performance than separate per-FPN-level masks, particularly for medium and large objects.
• SOLOv2 outperforms many box-based and box-free baselines on COCO and LVIS, with notable gains for large objects (e.g., AP_L improvements).
• Bounding-box byproducts derived from masks achieve competitive results, surpassing several traditional detectors in some configurations.