[Paper Review] ObjectLab: Automated Diagnosis of Mislabeled Images in Object Detection Data

Ulyana Tkachenko, Aditya Thyagarajan+1 more|arXiv (Cornell University)|Sep 2, 2023

Advanced Neural Network ApplicationsComputer Science3 citations

TL;DR

ObjectLab proposes a data-centric method to automatically detect and prioritize mislabeled images in object detection datasets by leveraging out-of-sample predictions from any trained detector. It identifies three error types—overlooked, badly located, and swapped labels—using similarity metrics and confidence scores, achieving superior precision and recall compared to existing label quality scoring methods across multiple datasets and models.

ABSTRACT

Despite powering sensitive systems like autonomous vehicles, object detection remains fairly brittle in part due to annotation errors that plague most real-world training datasets. We propose ObjectLab, a straightforward algorithm to detect diverse errors in object detection labels, including: overlooked bounding boxes, badly located boxes, and incorrect class label assignments. ObjectLab utilizes any trained object detection model to score the label quality of each image, such that mislabeled images can be automatically prioritized for label review/correction. Properly handling erroneous data enables training a better version of the same object detection model, without any change in existing modeling code. Across different object detection datasets (including COCO) and different models (including Detectron-X101 and Faster-RCNN), ObjectLab consistently detects annotation errors with much better precision/recall compared to other label quality scores.

Motivation & Objective

• To address the pervasive issue of annotation errors in real-world object detection datasets, which degrade model performance.
• To develop a general, model-agnostic method that identifies three types of label errors: overlooked, badly located, and swapped bounding boxes.
• To enable automatic prioritization of images for label review by assigning label quality scores based on model predictions.
• To improve object detection performance through data correction without modifying model architecture or training code.
• To provide a scalable, reusable solution for data-centric AI that remains effective across evolving models and training strategies.

Proposed method

• ObjectLab uses out-of-sample predictions from any trained object detector to score label quality for each image, ensuring no data leakage from training.
• It computes three distinct label quality scores per image: Overlooked (for missing objects), Badly Located (for imprecise bounding boxes), and Swapped (for incorrect class labels).
• For Badly Located scores, it measures IoU similarity between annotated and predicted boxes of the same class, with maximum score if no such prediction exists.
• For Swapped scores, it identifies high-confidence predictions (≥0.95) of a different class that are highly similar to the annotated box, assigning lower scores when such matches exist.
• For Overlooked errors, it evaluates high-confidence predictions (≥0.95) with no overlapping annotated box, using similarity and confidence to estimate likelihood of missing annotation.
• A Gaussian kernel is applied to avoid ties in similarity scores, with fixed hyperparameters α=σ=0.1, ensuring robustness to minor IoU variations.

Experimental results

Research questions

• RQ1 Can a model-agnostic, data-centric method effectively detect diverse label errors in object detection datasets without modifying model training?
• RQ2 How does ObjectLab’s label quality scoring compare to existing methods in terms of precision and recall for detecting overlooked, badly located, and swapped errors?
• RQ3 To what extent does using ObjectLab to prioritize label review improve the performance of the same object detection model without changing its architecture or training procedure?
• RQ4 Does ObjectLab maintain high performance across different datasets (e.g., COCO) and models (e.g., Faster R-CNN, Detectron-X101)?
• RQ5 Can ObjectLab’s approach be generalized to future models and training strategies, unlike model-specific noise-robust training techniques?

Key findings

• ObjectLab consistently outperforms existing label quality scoring methods in both precision and recall across multiple object detection datasets, including COCO.
• The method successfully identifies overlooked objects with high confidence, as demonstrated by a 99.8% confidence prediction for a missed fire hydrant in a COCO image.
• For swapped errors, ObjectLab detected a mislabeled glass as a bowl with a 96.2% confidence prediction of 'cup', highlighting its sensitivity to class label mismatches.
• In cases of badly located boxes, ObjectLab identified a partial table annotation with a 96.4% confidence prediction of a better-localized box, confirming its ability to detect spatial inaccuracies.
• The use of out-of-sample predictions and similarity-based scoring ensures robustness and generalization across different models and datasets without architectural or training code changes.
• ObjectLab’s approach enables direct data improvement—correcting mislabeled images leads to better model performance without altering model training procedures.