[Paper Review] Unifying Instance and Panoptic Segmentation with Dynamic Rank-1 Convolutions

Hao Chen, Chunhua Shen+1 more|arXiv (Cornell University)|Nov 19, 2020

Advanced Neural Network Applications27 references4 citations

TL;DR

This paper proposes DR1Mask, a unified, fully-convolutional panoptic segmentation framework that leverages dynamic rank-1 convolutions (DR1Conv) to efficiently fuse high-level context and low-level detail features, enabling state-of-the-art performance on both instance and panoptic segmentation with only a single additional classification layer. The method achieves 2x inference speedup over prior two-branch models while improving panoptic quality by 8 points in PQ on COCO using a ResNet-50 backbone.

ABSTRACT

Recently, fully-convolutional one-stage networks have shown superior performance comparing to two-stage frameworks for instance segmentation as typically they can generate higher-quality mask predictions with less computation. In addition, their simple design opens up new opportunities for joint multi-task learning. In this paper, we demonstrate that adding a single classification layer for semantic segmentation, fully-convolutional instance segmentation networks can achieve state-of-the-art panoptic segmentation quality. This is made possible by our novel dynamic rank-1 convolution (DR1Conv), a novel dynamic module that can efficiently merge high-level context information with low-level detailed features which is beneficial for both semantic and instance segmentation. Importantly, the proposed new method, termed DR1Mask, can perform panoptic segmentation by adding a single layer. To our knowledge, DR1Mask is the first panoptic segmentation framework that exploits a shared feature map for both instance and semantic segmentation by considering both efficacy and efficiency. Not only our framework is much more efficient -- twice as fast as previous best two-branch approaches, but also the unified framework opens up opportunities for using the same context module to improve the performance for both tasks. As a byproduct, when performing instance segmentation alone, DR1Mask is 10% faster and 1 point in mAP more accurate than previous state-of-the-art instance segmentation network BlendMask. Code is available at: https://git.io/AdelaiDet

Motivation & Objective

• To unify instance and panoptic segmentation into a single, efficient, fully-convolutional framework that shares features across tasks.
• To address the inefficiency and poor feature sharing in existing two-branch panoptic models, especially in semantic segmentation head design.
• To develop a dynamic feature fusion mechanism that is both parameter- and computation-efficient, even on high-dimensional feature maps.
• To enable high-resolution mask prediction with minimal inference cost while maintaining or improving accuracy.
• To demonstrate that a unified architecture can outperform separate models in both speed and accuracy for panoptic segmentation.

Proposed method

• Introduces DR1Conv, a dynamic rank-1 convolution module that computes second-order feature interactions between high-level and low-level features using low-rank approximation.
• Applies DR1Conv in intermediate layers to fuse context and detail features, enabling improved feature representation for both instance and semantic segmentation.
• Designs a unified network head where a single additional classification layer produces semantic segmentation output, eliminating the need for a separate 'stuff' branch.
• Employs tensor decomposition to compress the instance prediction head, adding nearly no computation while boosting mAP by 1%.
• Uses a shared feature map across instance and semantic heads, reducing redundancy and enabling joint optimization.
• Trains with a 3x schedule and multi-scale augmentation, with inference time measured under identical conditions for fair comparison.

Experimental results

Research questions

• RQ1 Can a unified, fully-convolutional architecture achieve state-of-the-art performance in both instance and panoptic segmentation with minimal computational overhead?
• RQ2 Can dynamic rank-1 convolutions effectively merge high-level context and low-level detail features across multiple tasks without increasing parameter or computation cost?
• RQ3 Does adding a single semantic head to an instance segmentation network yield competitive semantic segmentation performance while maintaining high inference speed?
• RQ4 How does the choice of basis feature dimensionality affect the performance of the unified semantic and instance segmentation heads?
• RQ5 Why does position-sensitive attention degrade panoptic segmentation performance, and how can this be mitigated in a unified framework?

Key findings

• DR1Mask achieves 44.5 mAP on COCO instance segmentation (test-dev) with only 79ms inference time using ResNet-50, outperforming BlendMask by 1 point mAP and 10% faster.
• On panoptic segmentation, DR1Mask achieves 42.9 PQ on COCO val2017 with 79ms inference, outperforming Panoptic-DeepLab by 8 PQ points and running twice as fast.
• The unified framework reduces inference time by 50% compared to previous best two-branch models like Panoptic-DeepLab, while improving PQ by 8 points.
• Doubling the basis feature dimension from 32 to 64 improves semantic segmentation quality by 2.1 points, highlighting the importance of channel width for class discrimination.
• Position-sensitive attention degrades semantic segmentation by 2.6 points, indicating that such mechanisms are suboptimal for unified panoptic learning.
• The proposed tensor decomposition-based head improves mAP by 1% with negligible computational cost, demonstrating efficiency gains in instance prediction.