[Paper Review] Active learning with MaskAL reduces annotation effort for training Mask R-CNN

Pieter M. Blok, Gert Kootstra+4 more|arXiv (Cornell University)|Dec 13, 2021

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes MaskAL, an active learning method for Mask R-CNN that reduces annotation effort by iteratively selecting the most uncertain (hard-to-classify) instances from unlabelled data. It achieves 93.9% of full-dataset Mask R-CNN performance using only 17.9% of the training data—outperforming random sampling, which required 2300 images to match MaskAL’s performance at 900 images.

ABSTRACT

The generalisation performance of a convolutional neural network (CNN) is influenced by the quantity, quality, and variety of the training images. Training images must be annotated, and this is time consuming and expensive. The goal of our work was to reduce the number of annotated images needed to train a CNN while maintaining its performance. We hypothesised that the performance of a CNN can be improved faster by ensuring that the set of training images contains a large fraction of hard-to-classify images. The objective of our study was to test this hypothesis with an active learning method that can automatically select the hard-to-classify images. We developed an active learning method for Mask Region-based CNN (Mask R-CNN) and named this method MaskAL. MaskAL involved the iterative training of Mask R-CNN, after which the trained model was used to select a set of unlabelled images about which the model was most uncertain. The selected images were then annotated and used to retrain Mask R-CNN, and this was repeated for a number of sampling iterations. In our study, MaskAL was compared to a random sampling method on a broccoli dataset with five visually similar classes. MaskAL performed significantly better than the random sampling. In addition, MaskAL had the same performance after sampling 900 images as the random sampling had after 2300 images. Compared to a Mask R-CNN model that was trained on the entire training set (14,000 images), MaskAL achieved 93.9% of that model's performance with 17.9% of its training data. The random sampling achieved 81.9% of that model's performance with 16.4% of its training data. We conclude that by using MaskAL, the annotation effort can be reduced for training Mask R-CNN on a broccoli dataset with visually similar classes. Our software is available on https://github.com/pieterblok/maskal.

Motivation & Objective

• Reduce the annotation effort required to train Mask R-CNN without sacrificing model performance.
• Address the high cost and time consumption of annotating instance segmentation data in agriculture.
• Test the hypothesis that selecting hard-to-classify (uncertain) samples accelerates model convergence.
• Develop and evaluate an active learning framework tailored for instance segmentation with Mask R-CNN.
• Demonstrate that uncertainty-based sampling outperforms random sampling in data-efficient training.

Proposed method

• Iteratively train Mask R-CNN on a growing pool of annotated images.
• Use the trained model to predict segmentation masks on unlabelled images and compute uncertainty scores based on prediction confidence.
• Select unlabelled images with the highest uncertainty (lowest confidence) for human annotation.
• Annotate the selected images and retrain the Mask R-CNN model with the expanded dataset.
• Repeat the process over multiple sampling iterations to refine model performance.
• Use a confidence threshold based on the model’s predicted mask quality to identify uncertain samples.

Experimental results

Research questions

• RQ1 Can active learning based on model uncertainty reduce the number of annotated images needed to train a Mask R-CNN model?
• RQ2 Does selecting hard-to-classify instances lead to faster convergence and better performance compared to random sampling?
• RQ3 How does the performance of a Mask R-CNN model trained with MaskAL compare to one trained on the full dataset?
• RQ4 What is the data efficiency gain of MaskAL over random sampling on a visually complex agricultural dataset?
• RQ5 Can uncertainty-based sampling maintain high performance with significantly fewer training annotations?

Key findings

• MaskAL achieved 93.9% of the performance of a Mask R-CNN model trained on the full dataset (14,000 images) using only 17.9% of the data (2,500 images).
• The random sampling baseline required 2,300 images to reach the same performance level as MaskAL with only 900 images.
• MaskAL outperformed random sampling by achieving higher mean average precision (mAP) at every data point during training.
• The performance gain of MaskAL was most pronounced in early training stages, indicating faster learning from informative samples.
• The method significantly reduced annotation effort while maintaining high segmentation accuracy on a broccoli dataset with five visually similar classes.
• The source code for MaskAL is publicly available on GitHub, enabling reproducibility and further research.