[Paper Review] Synthetic Data and Hierarchical Object Detection in Overhead Imagery

Nathan Clement, Alan Schoen+2 more|arXiv (Cornell University)|Jan 29, 2021

Remote-Sensing Image Classification4 citations

TL;DR

This paper proposes a hierarchical object detection framework using synthetic data to improve performance in low- and zero-shot remote sensing scenarios. It introduces the GAN-Reskinner for realistic reskinned 3D-rendered imagery and a broad-to-narrow detection architecture that leverages real data for broad classes and synthetic data for narrow target classes, achieving state-of-the-art recall in zero-shot settings on overhead satellite imagery.

ABSTRACT

The performance of neural network models is often limited by the availability of big data sets. To treat this problem, we survey and develop novel synthetic data generation and augmentation techniques for enhancing low/zero-sample learning in satellite imagery. In addition to extending synthetic data generation approaches, we propose a hierarchical detection approach to improve the utility of synthetic training samples. We consider existing techniques for producing synthetic imagery--3D models and neural style transfer--as well as introducing our own adversarially trained reskinning network, the GAN-Reskinner, to blend 3D models. Additionally, we test the value of synthetic data in a two-stage, hierarchical detection/classification model of our own construction. To test the effectiveness of synthetic imagery, we employ it in the training of detection models and our two stage model, and evaluate the resulting models on real satellite images. All modalities of synthetic data are tested extensively on practical, geospatial analysis problems. Our experiments show that synthetic data developed using our approach can often enhance detection performance, particularly when combined with some real training images. When the only source of data is synthetic, our GAN-Reskinner often boosts performance over conventionally rendered 3D models and in all cases the hierarchical model outperforms the baseline end-to-end detection architecture.

Motivation & Objective

• Address the challenge of limited real training data for object detection in overhead satellite imagery, particularly in low- and zero-shot learning regimes.
• Reduce the domain gap between synthetic and real satellite imagery by improving the realism of synthetic training samples.
• Develop a modular, two-stage detection architecture that separates detection and classification for better generalization and data efficiency.
• Evaluate the effectiveness of multiple synthetic data generation techniques—3D CAD rendering, GAN-Reskinner, and neural style transfer—on real-world geospatial detection tasks.
• Demonstrate that hierarchical detection using synthetic data significantly improves recall and precision, especially when real data for the target class is absent.

Proposed method

• Propose the GAN-Reskinner, a conditional GAN-based reskinning network that takes 3D-rendered synthetic images and re-renders them with realistic textures to reduce domain shift.
• Integrate 3D CAD models with adversarial training to generate diverse, realistic synthetic overhead imagery by learning to transfer textures from real images to synthetic objects.
• Apply neural style transfer to augment small real datasets, improving model robustness to variations in imaging conditions such as seasonality and lighting.
• Design a broad-to-narrow two-stage detection architecture: first, a broad detector identifies general object categories using real data; second, a narrow classifier uses synthetic data to refine predictions for specific target classes.
• Train the broad detector on real data from related classes (e.g., all vehicles), while the narrow classifier is trained exclusively on synthetic data for the target class (e.g., compact passenger cars).
• Use a joint training strategy where the broad detector’s outputs are filtered by the narrow classifier, enabling end-to-end optimization with synthetic data for fine-grained classification.

Experimental results

Research questions

• RQ1 Can synthetic data generated via 3D CAD rendering and GAN-based reskinned textures improve object detection performance in overhead satellite imagery compared to real data alone?
• RQ2 How does the GAN-Reskinner technique reduce the domain gap between synthetic and real satellite imagery in object detection tasks?
• RQ3 To what extent does the broad-to-narrow hierarchical detection architecture improve zero-shot and low-shot detection performance compared to end-to-end models?
• RQ4 How effective is neural style transfer in augmenting small real datasets to improve model generalization across diverse imaging conditions?
• RQ5 Does combining multiple synthetic data generation techniques (3D rendering, GAN-Reskinner, style transfer) yield better detection performance than any single method?

Key findings

• The GAN-Reskinner consistently improves detection performance over conventionally rendered 3D models, particularly in reducing domain shift and enhancing feature discrimination.
• In zero-shot detection, the broad-to-narrow model achieved 100% recall for aircraft and passenger vehicles when trained with real data for broad classes and synthetic data for narrow classes, outperforming end-to-end models.
• The broad-to-narrow architecture achieved 97.6% recall for aircraft and 97.0% for passenger vehicles in low-shot settings with synthetic-only training for the narrow class, significantly exceeding end-to-end baselines.
• When only synthetic data was used, the GAN-Reskinner-based model achieved 100% recall for aircraft and 96.3% for passenger vehicles, demonstrating its effectiveness in zero-shot regimes.
• The hierarchical model reduced false positives by filtering broad detections through a narrow classifier, with the best configuration (R+C+G+N) achieving 99.4% recall for aircraft and 95.1% for passenger vehicles with minimal false detections.
• Neural style transfer improved model robustness to imaging condition variations, and when combined with other synthetic data techniques, enhanced generalization across different seasons and lighting conditions.