[Paper Review] ConsInstancy: Learning Instance Representations for Semi-Supervised Panoptic Segmentation of Concrete Aggregate Particles

Max Coenen, Tobias Schack+3 more|arXiv (Cornell University)|Apr 10, 2022

Infrastructure Maintenance and Monitoring24 references17 citations

TL;DR

This paper proposes ConsInstancy, a semi-supervised learning framework for panoptic segmentation of concrete aggregate particles using novel instance representations—3D orientation maps and complementary distance transform maps—predicted via a single forward pass through a fully convolutional network. By enforcing consistency between predicted instance representations and semantic segmentations on unlabelled data, the method boosts performance, achieving up to 5% higher overall accuracy than fully supervised training and outperforming state-of-the-art semi-supervised methods by up to 1.5% on two challenging datasets, including a new fresh concrete dataset.

ABSTRACT

We present a semi-supervised method for panoptic segmentation based on ConsInstancy regularisation, a novel strategy for semi-supervised learning. It leverages completely unlabelled data by enforcing consistency between predicted instance representations and semantic segmentations during training in order to improve the segmentation performance. To this end, we also propose new types of instance representations that can be predicted by one simple forward path through a fully convolutional network (FCN), delivering a convenient and simple-to-train framework for panoptic segmentation. More specifically, we propose the prediction of a three-dimensional instance orientation map as intermediate representation and two complementary distance transform maps as final representation, providing unique instance representations for a panoptic segmentation. We test our method on two challenging data sets of both, hardened and fresh concrete, the latter being proposed by the authors in this paper demonstrating the effectiveness of our approach, outperforming the results achieved by state-of-the-art methods for semi-supervised segmentation. In particular, we are able to show that by leveraging completely unlabeled data in our semi-supervised approach the achieved overall accuracy (OA) is increased by up to 5% compared to an entirely supervised training using only labeled data. Furthermore, we exceed the OA achieved by state-of-the-art semi-supervised methods by up to 1.5%.

Motivation & Objective

• To address the high annotation cost of instance-aware segmentation in concrete imagery, especially for dense, small particles.
• To improve panoptic segmentation performance with limited labelled data by leveraging abundant unlabelled data in a semi-supervised manner.
• To introduce new, learnable instance representations that enable end-to-end training with a single forward pass.
• To demonstrate the effectiveness of the method on both hardened and fresh concrete datasets, including a novel fresh concrete dataset.
• To outperform existing semi-supervised and supervised state-of-the-art methods in panoptic segmentation of concrete aggregates.

Proposed method

• Proposes a 3D instance orientation map as an intermediate representation to encode spatial instance structure.
• Introduces two complementary distance transform maps (Iδ+ and Iδ−) as final instance representations for unique instance identification.
• Employs a multi-task fully convolutional network (FCN) with shared encoder and separate decoders for semantic and instance representations.
• Applies ConsInstancy regularization, enforcing consistency between predicted instance representations and semantic segmentations on unlabelled data during training.
• Uses a consistency loss that aligns the predicted instance maps with the semantic segmentation output, improving generalization.
• Trains the network using both labelled data for supervision and unlabelled data for consistency regularization, reducing reliance on annotations.

Experimental results

Research questions

• RQ1 Can novel instance representations enable effective and efficient panoptic segmentation in a single forward pass?
• RQ2 How effective is ConsInstancy regularization in leveraging unlabelled data to improve panoptic segmentation performance?
• RQ3 Can the proposed method achieve superior performance compared to state-of-the-art semi-supervised and fully supervised methods on concrete aggregate segmentation?
• RQ4 Does the method generalize to challenging data distributions, such as fresh concrete with indistinct boundaries?
• RQ5 To what extent does semi-supervised learning reduce the need for expensive instance-level annotations in concrete imagery?

Key findings

• The ConsInstancy method achieves a 5% increase in overall accuracy (OA) compared to fully supervised training using only labelled data.
• On the sedimentation dataset, the method improves mean F1 score for instance segmentation by 2.6% and for semantic segmentation by 1.9%.
• On the fresh concrete dataset, the ConsInstancy variant achieves a PQ score of 39.0% and F inst1 score of 52.6%, outperforming the best supervised baseline by 8.0% and 9.4% respectively.
• The ConsInstancy variant reduces false positive predictions on the aggregate class, as evidenced by significantly improved precision scores.
• The method outperforms state-of-the-art semi-supervised methods by up to 1.5% in overall accuracy, particularly excelling on the fresh concrete dataset.
• Qualitative results show smoother instance boundaries and fewer erroneous segmentation artifacts in the ConsInstancy variant compared to supervised baselines.