[论文解读] Active learning with MaskAL reduces annotation effort for training Mask R-CNN

Pieter M. Blok, Gert Kootstra+4 more|arXiv (Cornell University)|Dec 13, 2021

Generative Adversarial Networks and Image Synthesis被引用 4

一句话总结

该论文提出 MaskAL，一种用于 Mask R-CNN 的主动学习方法，通过迭代选择未标注数据中最具不确定性的（难以分类的）样本，减少标注工作量。仅使用 17.9% 的训练数据（即 2,500 张图像），MaskAL 即可达到完整数据集 Mask R-CNN 性能的 93.9%，优于随机采样方法——后者需 2,300 张图像才能达到 MaskAL 在 900 张图像时的性能水平。

ABSTRACT

The generalisation performance of a convolutional neural network (CNN) is influenced by the quantity, quality, and variety of the training images. Training images must be annotated, and this is time consuming and expensive. The goal of our work was to reduce the number of annotated images needed to train a CNN while maintaining its performance. We hypothesised that the performance of a CNN can be improved faster by ensuring that the set of training images contains a large fraction of hard-to-classify images. The objective of our study was to test this hypothesis with an active learning method that can automatically select the hard-to-classify images. We developed an active learning method for Mask Region-based CNN (Mask R-CNN) and named this method MaskAL. MaskAL involved the iterative training of Mask R-CNN, after which the trained model was used to select a set of unlabelled images about which the model was most uncertain. The selected images were then annotated and used to retrain Mask R-CNN, and this was repeated for a number of sampling iterations. In our study, MaskAL was compared to a random sampling method on a broccoli dataset with five visually similar classes. MaskAL performed significantly better than the random sampling. In addition, MaskAL had the same performance after sampling 900 images as the random sampling had after 2300 images. Compared to a Mask R-CNN model that was trained on the entire training set (14,000 images), MaskAL achieved 93.9% of that model's performance with 17.9% of its training data. The random sampling achieved 81.9% of that model's performance with 16.4% of its training data. We conclude that by using MaskAL, the annotation effort can be reduced for training Mask R-CNN on a broccoli dataset with visually similar classes. Our software is available on https://github.com/pieterblok/maskal.

研究动机与目标

• 在不牺牲模型性能的前提下，减少训练 Mask R-CNN 所需的标注工作量。
• 解决农业领域实例分割数据标注成本高、耗时长的问题。
• 检验假设：选择难以分类（不确定）的样本可加速模型收敛。
• 开发并评估一种专用于实例分割的主动学习框架，适用于 Mask R-CNN。
• 证明基于不确定性的采样方法在数据高效训练中优于随机采样。

提出的方法

• 在逐步扩大的已标注图像池上迭代训练 Mask R-CNN。
• 利用训练好的模型对未标注图像进行分割掩码预测，并基于预测置信度计算不确定性分数。
• 选择不确定性最高（置信度最低）的未标注图像进行人工标注。
• 对选定图像进行标注，并使用扩展后的数据集重新训练 Mask R-CNN 模型。
• 在多个采样迭代中重复该过程，以优化模型性能。
• 基于模型预测的掩码质量置信度阈值，识别不确定性样本。

实验结果

研究问题

• RQ1 基于模型不确定性的主动学习是否能减少训练 Mask R-CNN 所需的标注图像数量？
• RQ2 与随机采样相比，选择难以分类的样本是否能实现更快的收敛速度和更高的性能？
• RQ3 使用 MaskAL 训练的 Mask R-CNN 模型性能与在完整数据集上训练的模型相比如何？
• RQ4 在视觉复杂的农业数据集上，MaskAL 相较于随机采样在数据效率方面有何提升？
• RQ5 基于不确定性的采样方法是否能在显著减少训练标注数量的同时保持高性能？

主要发现

• MaskAL 仅使用 17.9% 的数据（即 2,500 张图像），即达到完整数据集（14,000 张图像）训练的 Mask R-CNN 模型 93.9% 的性能。
• 随机采样基线方法需 2,300 张图像才能达到 MaskAL 在仅 900 张图像时的性能水平。
• 在训练的每个数据点上，MaskAL 均实现了比随机采样更高的平均精度均值（mAP）。
• MaskAL 的性能提升在训练初期最为显著，表明其能从信息量丰富的样本中更快学习。
• 该方法显著减少了标注工作量，同时在包含五个视觉相似类别的西兰花数据集上保持了高分割精度。
• MaskAL 的源代码已公开发布于 GitHub，支持可复现性与进一步研究。