[Paper Review] Quantity beats quality for semantic segmentation of corrosion in images

Will Nash, Tom Drummond+1 more|arXiv (Cornell University)|Jun 30, 2018

Non-Destructive Testing Techniques1 references3 citations

TL;DR

This study demonstrates that training a fully convolutional network (FCN) on a large, noisy dataset of 250 corrosion images labeled by undergraduates outperforms training on a small, expert-annotated dataset of 10 images, despite the latter's higher labeling quality. The key finding is that data quantity outweighs quality in semantic segmentation for complex, specialist subjects like corrosion, where model robustness to noise is enhanced by sufficient training examples, leading to better generalization and lower overfitting than with limited expert-labeled data.

ABSTRACT

Dataset creation is typically one of the first steps when applying Artificial Intelligence methods to a new task; and the real world performance of models hinges on the quality and quantity of data available. Producing an image dataset for semantic segmentation is resource intensive, particularly for specialist subjects where class segmentation is not able to be effectively farmed out. The benefit of producing a large, but poorly labelled, dataset versus a small, expertly segmented dataset for semantic segmentation is an open question. Here we show that a large, noisy dataset outperforms a small, expertly segmented dataset for training a Fully Convolutional Network model for semantic segmentation of corrosion in images. A large dataset of 250 images with segmentations labelled by undergraduates and a second dataset of just 10 images, with segmentations labelled by subject matter experts were produced. The mean Intersection over Union and micro F-score metrics were compared after training for 50,000 epochs. This work is illustrative for researchers setting out to develop deep learning models for detection and location of specialist features.

Motivation & Objective

• To investigate whether a large, noisy dataset or a small, expert-annotated dataset yields better performance in semantic segmentation of corrosion in images.
• To evaluate the trade-off between data quality and data quantity in training deep learning models for specialist visual tasks.
• To determine if label noise from non-expert labeling significantly degrades model performance compared to high-quality expert annotations.
• To assess the feasibility of using non-expert labeling for creating large-scale datasets in specialized domains like corrosion detection.
• To propose a practical training strategy for semantic segmentation in low-resource, domain-specific scenarios.

Proposed method

• Two datasets were created: DS-A with 250 images segmented by undergraduates (low-quality labels), and DS-B with 10 images segmented by corrosion experts (high-quality, multi-class labels).
• A fully convolutional network (FCN) was trained on both datasets using strong supervision with pixel-wise annotations.
• Training was conducted for 50,000 epochs on both datasets, with performance evaluated using mean Intersection over Union (mIoU) and micro F-score metrics.
• The model trained on DS-A was compared to the model trained on DS-B to assess generalization and robustness to label noise.
• A multi-class model was trained on DS-B to evaluate performance on minor, moderate, and severe corrosion classes.
• Fine-tuning was suggested as a follow-up strategy to improve discrimination and accuracy after pre-training on the large noisy dataset.

Experimental results

Research questions

• RQ1 Does a large dataset with noisy, non-expert annotations outperform a small, expert-annotated dataset in semantic segmentation of corrosion images?
• RQ2 To what extent does label noise from non-expert labeling degrade model performance compared to high-quality annotations?
• RQ3 Can a deep learning model generalize effectively from a large number of imperfectly labeled examples, even when boundaries are ambiguous?
• RQ4 Is overfitting a significant issue when training on a small expert-annotated dataset with limited diversity?
• RQ5 What is the optimal training strategy for semantic segmentation in specialist domains with scarce expert resources?

Key findings

• The model trained on the large, noisy dataset (DS-A) achieved a mean Intersection over Union (mIoU) of 0.29 and a micro F-score of 0.23, outperforming the model trained on the small expert dataset (DS-B).
• The DS-B model exhibited significant overfitting, particularly on Image 3, where performance dropped sharply, indicating poor generalization due to limited training data.
• The DS-A model showed better robustness to label noise, as the sheer volume of training examples allowed the model to learn generalizable features despite inaccuracies in segmentation boundaries.
• The total number of labeled pixels in DS-B was only 27 million, which was insufficient to train a model with 134 million parameters without overfitting.
• The DS-B model achieved a mIoU of 0.26 and micro F-score of 0.27 on the validation set, significantly lower than the DS-A model’s metrics.
• The study concludes that pre-training on a large, noisy dataset followed by fine-tuning on a small expert-annotated dataset is a viable and effective strategy for improving accuracy and class discrimination in corrosion segmentation.