[Paper Review] Metamorphic Testing for Object Detection Systems

Shuai Wang, Zhendong Su|arXiv (Cornell University)|Dec 19, 2019

Advanced Neural Network Applications78 references17 citations

TL;DR

This paper introduces MetaOD, the first metamorphic testing framework for object detection systems, which generates natural-looking synthetic images by inserting objects into backgrounds and checks consistency in detection results between original and modified images. It uncovered tens of thousands of defects in commercial detectors and improved model accuracy from mAP 9.3 to 10.5 via retraining with failure-triggering synthetic data.

ABSTRACT

Recent advances in deep neural networks (DNNs) have led to object detectors that can rapidly process pictures or videos, and recognize the objects that they contain. Despite the promising progress by industrial manufacturers such as Amazon and Google in commercializing deep learning-based object detection as a standard computer vision service, object detection systems - similar to traditional software - may still produce incorrect results. These errors, in turn, can lead to severe negative outcomes for the users of these object detection systems. For instance, an autonomous driving system that fails to detect pedestrians can cause accidents or even fatalities. However, principled, systematic methods for testing object detection systems do not yet exist, despite their importance. To fill this critical gap, we introduce the design and realization of MetaOD, the first metamorphic testing system for object detectors to effectively reveal erroneous detection results by commercial object detectors. To this end, we (1) synthesize natural-looking images by inserting extra object instances into background images, and (2) design metamorphic conditions asserting the equivalence of object detection results between the original and synthetic images after excluding the prediction results on the inserted objects. MetaOD is designed as a streamlined workflow that performs object extraction, selection, and insertion. Evaluated on four commercial object detection services and four pretrained models provided by the TensorFlow API, MetaOD found tens of thousands of detection defects in these object detectors. To further demonstrate the practical usage of MetaOD, we use the synthetic images that cause erroneous detection results to retrain the model. Our results show that the model performance is increased significantly, from an mAP score of 9.3 to an mAP score of 10.5.

Motivation & Objective

• To address the critical lack of systematic testing methods for deep learning-based object detection systems, which can lead to severe safety risks in applications like autonomous driving.
• To develop a practical, black-box testing approach that reveals erroneous detection results without requiring access to model internals.
• To generate diverse, natural-looking synthetic images that preserve global and local realism to effectively trigger edge-case prediction failures.
• To demonstrate that synthetic images exposing defects can be used to retrain models and improve their robustness and accuracy.
• To establish a novel, reusable testing pipeline for commercial and pretrained object detectors using metamorphic testing principles.

Proposed method

• Synthesizes natural-looking images by inserting object instances into background images using instance segmentation to extract high-quality object samples.
• Applies lightweight, domain-aware criteria to select and refine objects based on visual and contextual compatibility with the background.
• Employs delta debugging-inspired heuristics to identify insertion locations that maximize the likelihood of triggering detection errors while preserving image realism.
• Defines metamorphic conditions asserting that detection results on original objects should remain consistent between original and synthetic images, excluding predictions on inserted objects.
• Automates a three-stage workflow: object extraction, refinement/selection, and context-aware insertion to generate diverse test inputs.
• Uses the detected inconsistencies as a failure oracle to identify defective behavior in commercial and open-source object detectors.

Experimental results

Research questions

• RQ1 Can metamorphic testing be effectively applied to detect defects in commercial object detection systems where test oracles are unavailable?
• RQ2 How can synthetic images be generated to preserve both local realism of inserted objects and global scene consistency to trigger meaningful errors?
• RQ3 To what extent can synthetic images that expose detection defects be used to improve model robustness through retraining?
• RQ4 What are the key factors in object selection and insertion that maximize the detection of edge-case failures in object detectors?
• RQ5 How does the proposed method compare in defect detection capability to existing testing techniques for deep learning systems?

Key findings

• MetaOD detected tens of thousands of erroneous detection results across four commercial object detection services (Amazon, Google, IBM, Microsoft) and four TensorFlow API models.
• The framework successfully identified defects in real-world systems, including failures to detect objects under challenging visual conditions, such as low-contrast or occluded scenarios.
• Retraining a model using synthetic images that triggered detection errors improved its mean average precision (mAP) from 9.3 to 10.5, demonstrating the effectiveness of failure-aware retraining.
• The synthetic images generated by MetaOD preserved high levels of realism, enabling the detection of subtle, context-sensitive errors that standard data augmentation techniques often miss.
• The metamorphic testing approach proved effective even in a black-box setting, requiring no access to model weights or internal architecture.
• The method’s lightweight, modular pipeline enables scalable and repeatable testing of commercial object detection systems without requiring domain-specific fine-tuning.