[Paper Review] Instance and Panoptic Segmentation Using Conditional Convolutions

Zhi Tian, Bowen Zhang+2 more|arXiv (Cornell University)|Feb 5, 2021

Advanced Neural Network Applications48 references4 citations

TL;DR

This paper proposes CondInst, a fully convolutional framework for instance and panoptic segmentation that replaces ROI-based feature cropping with dynamic, instance-aware conditional convolutions. By generating mask head filters conditionally per instance, it achieves state-of-the-art accuracy on COCO with faster inference and higher mask resolution, outperforming Mask R-CNN and prior methods in both speed and quality without requiring longer training schedules.

ABSTRACT

We propose a simple yet effective framework for instance and panoptic segmentation, termed CondInst (conditional convolutions for instance and panoptic segmentation). In the literature, top-performing instance segmentation methods typically follow the paradigm of Mask R-CNN and rely on ROI operations (typically ROIAlign) to attend to each instance. In contrast, we propose to attend to the instances with dynamic conditional convolutions. Instead of using instance-wise ROIs as inputs to the instance mask head of fixed weights, we design dynamic instance-aware mask heads, conditioned on the instances to be predicted. CondInst enjoys three advantages: 1.) Instance and panoptic segmentation are unified into a fully convolutional network, eliminating the need for ROI cropping and feature alignment. 2.) The elimination of the ROI cropping also significantly improves the output instance mask resolution. 3.) Due to the much improved capacity of dynamically-generated conditional convolutions, the mask head can be very compact (e.g., 3 conv. layers, each having only 8 channels), leading to significantly faster inference time per instance and making the overall inference time almost constant, irrelevant to the number of instances. We demonstrate a simpler method that can achieve improved accuracy and inference speed on both instance and panoptic segmentation tasks. On the COCO dataset, we outperform a few state-of-the-art methods. We hope that CondInst can be a strong baseline for instance and panoptic segmentation. Code is available at: https://git.io/AdelaiDet

Motivation & Objective

• To address the limitations of ROI-based instance segmentation, such as variable inference time and low mask resolution due to fixed-size cropping and feature alignment.
• To unify instance and panoptic segmentation into a fully convolutional framework, eliminating the need for ROI cropping and ROIAlign.
• To improve mask quality and inference efficiency by using compact, dynamic filter generation that adapts to each instance.
• To demonstrate that a simpler, fully convolutional design with conditional convolutions can outperform strong two-stage baselines like Mask R-CNN in both accuracy and speed.

Proposed method

• CondInst replaces the fixed-weight mask head in Mask R-CNN with a dynamic, instance-aware mask head that generates filters conditionally based on the target instance embedding.
• The mask head uses a lightweight backbone (e.g., 3 conv layers with only 8 channels each) to predict filter weights for each instance using a condition vector derived from the instance's feature representation.
• Conditional convolutions are implemented via a lightweight network that predicts the kernel weights of the mask head convolutions based on the instance's RoI feature, enabling instance-specific filtering.
• The framework operates end-to-end in a fully convolutional manner, eliminating ROI cropping and feature alignment, and directly outputs high-resolution masks per instance.
• For panoptic segmentation, a separate semantic segmentation head is attached to the same feature maps, with instance masks reweighted to ensure class consistency and avoid overlap.
• The method is trained using standard instance segmentation and panoptic segmentation targets, with mask annotations adapted to panoptic labeling rules (e.g., overlapping masks assigned to the foreground instance only).

Experimental results

Research questions

• RQ1 Can a fully convolutional network with dynamic, instance-aware convolution filters outperform ROI-based methods like Mask R-CNN in instance segmentation accuracy and inference speed?
• RQ2 How does eliminating ROI cropping and feature alignment affect mask resolution and model efficiency in instance segmentation?
• RQ3 Can a compact, dynamic mask head with only 8-channel convolutions achieve competitive performance compared to larger, fixed-weight heads?
• RQ4 Does the use of conditional convolutions enable better generalization and detail preservation in instance masks compared to standard FCNs?
• RQ5 Can the same framework be effectively extended to panoptic segmentation with minimal modifications and achieve state-of-the-art performance?

Key findings

• CondInst achieves 47.8 PQ on COCO test-dev using a ResNeXt-101-FPN backbone with 3× training schedule, outperforming Panoptic-FPN (46.1 PQ) and AdaptIS (42.8 PQ).
• On Cityscapes val set, CondInst achieves 61.7 PQ, surpassing previous methods including Unifying (61.4 PQ) and Panoptic-FCN (61.4 PQ).
• The method achieves higher mask resolution than ROI-based methods because it avoids resizing cropped ROIs to fixed sizes.
• Inference time per instance is significantly faster and less dependent on instance count due to the compact, dynamic mask head architecture.
• CondInst achieves state-of-the-art performance on both instance and panoptic segmentation benchmarks without requiring longer training schedules or complex post-processing.
• Qualitative results show that CondInst produces higher-quality masks with better preservation of fine details compared to YOLACT and Mask R-CNN.