[Paper Review] How much real data do we actually need: Analyzing object detection performance using synthetic and real data

Farzan Erlik Nowruzi, Prince Kapoor+4 more|arXiv (Cornell University)|Jul 16, 2019

Advanced Neural Network Applications46 references67 citations

TL;DR

The paper analyzes how synthetic data can substitute or complement real data for SSD-MobileNet object detection in autonomous driving, and proposes training strategies (mixed training and fine-tuning) to maximize performance with limited real data.

ABSTRACT

In recent years, deep learning models have resulted in a huge amount of progress in various areas, including computer vision. By nature, the supervised training of deep models requires a large amount of data to be available. This ideal case is usually not tractable as the data annotation is a tremendously exhausting and costly task to perform. An alternative is to use synthetic data. In this paper, we take a comprehensive look into the effects of replacing real data with synthetic data. We further analyze the effects of having a limited amount of real data. We use multiple synthetic and real datasets along with a simulation tool to create large amounts of cheaply annotated synthetic data. We analyze the domain similarity of each of these datasets. We provide insights about designing a methodological procedure for training deep networks using these datasets.

Motivation & Objective

• Assess the impact of reducing real data on object detection performance.
• Study the relationship between dataset distribution/diversity and model accuracy.
• Evaluate mixed training (synthetic and real) versus fine-tuning with real data.
• Propose procedural guidance for training with large synthetic and small real datasets.
• Investigate effects of combining multiple synthetic datasets on real-data generalization.

Proposed method

• Train SSD-MobileNet on a mix of real and synthetic datasets across car and person classes.
• Systematically vary real data fractions (100%, 10%, 5%, 2.5%) and synthetic-to-real ratios.
• Evaluate using precision/recall over IoU thresholds 0.5–0.95 with 0.05 steps.
• Perform transfer learning by pretraining on synthetic data then fine-tuning on real data.
• Compare mixed training versus fine-tuning and dataset combinations.

Experimental results

Research questions

• RQ1 How does reducing real training data affect object detection performance across datasets?
• RQ2 How do synthetic datasets differ in their ability to train models that generalize to real data?
• RQ3 Does mixing synthetic with real data improve performance over real-data-only training at equivalent real-data levels?
• RQ4 Is fine-tuning on real data after synthetic pretraining more effective than mixed training?
• RQ5 What is the impact of combining multiple synthetic datasets on real-data generalization?

Key findings

• Reducing real data substantially degrades precision and recall for both car and person detection.
• Person detection is more sensitive to data reduction than car detection due to deformability and variety.
• Mixed training with synthetic data improves performance over using only a fraction of real data, but fine-tuning after synthetic pretraining yields larger gains, especially for the person class.
• Transfer learning (synthetic pretraining followed by real-data fine-tuning) provides more stable gains and better recall than mixed training.
• Combining multiple synthetic datasets generally improves results over single synthetic sources, except for some real datasets like BDD.
• 7D (pure synthetic) is well matched to real data after fine-tuning, while CARLA underperforms due to less accurate environment/camera models.