[Paper Review] Whale Detection Enhancement through Synthetic Satellite Images

Akshaj Gaur, Cheng Liu+3 more|arXiv (Cornell University)|Aug 15, 2023

Marine animal studies overviewEnvironmental Science3 citations

TL;DR

This paper introduces SeaDroneSim2, a synthetic satellite and aerial image generation platform that enhances whale detection in marine environments by augmenting real datasets with photorealistic synthetic data. By training object detection models on just 10% real data augmented with synthetic images, the authors achieve a 15.4% improvement in detection recall (DR50) and a 15.7% gain in DR60, demonstrating significant performance gains over real-data-only training.

ABSTRACT

With a number of marine populations in rapid decline, collecting and analyzing data about marine populations has become increasingly important to develop effective conservation policies for a wide range of marine animals, including whales. Modern computer vision algorithms allow us to detect whales in images in a wide range of domains, further speeding up and enhancing the monitoring process. However, these algorithms heavily rely on large training datasets, which are challenging and time-consuming to collect particularly in marine or aquatic environments. Recent advances in AI however have made it possible to synthetically create datasets for training machine learning algorithms, thus enabling new solutions that were not possible before. In this work, we present a solution - SeaDroneSim2 benchmark suite, which addresses this challenge by generating aerial, and satellite synthetic image datasets to improve the detection of whales and reduce the effort required for training data collection. We show that we can achieve a 15% performance boost on whale detection compared to using the real data alone for training, by augmenting a 10% real data. We open source both the code of the simulation platform SeaDroneSim2 and the dataset generated through it.

Motivation & Objective

• To address the scarcity and high cost of real-world marine imaging data for whale detection.
• To reduce reliance on expensive and time-consuming manual data collection and annotation in marine environments.
• To develop a scalable, customizable simulation platform that generates diverse, photorealistic synthetic aerial and satellite images for training object detection models.
• To evaluate the effectiveness of synthetic data in improving detection performance across different network architectures and data ratios.
• To open-source both the SeaDroneSim2 simulation framework and the generated synthetic dataset to accelerate research in marine object detection.

Proposed method

• Leveraged Blender™ as a rendering engine to build a physics-based simulation environment for maritime scenes.
• Designed customizable virtual scenes with adjustable parameters including water properties, lighting, camera altitude, viewing angles, and noise levels.
• Generated photorealistic synthetic aerial and satellite images of whales and other marine objects with precise instance segmentation masks.
• Integrated synthetic data with real datasets in varying ratios (10%, 50%) to train and fine-tune object detection models.
• Evaluated performance using DR50 and DR60 metrics—measures of detection recall at 50% and 60% overlap thresholds—against human-annotated ground truth.
• Compared detection performance across different models (Unet, FPN) and data combinations to isolate the impact of synthetic data augmentation.

Experimental results

Research questions

• RQ1 Can synthetic satellite and aerial images significantly improve whale detection performance when used to augment limited real datasets?
• RQ2 How does the inclusion of synthetic data affect detection recall (DR50 and DR60) compared to training on real data alone?
• RQ3 To what extent does synthetic data mitigate false negatives in whale detection, especially for partially submerged or low-visibility cases?
• RQ4 How does the performance of detection models trained on mixed real-synthetic data compare to models trained on larger real-data subsets?
• RQ5 Can a simulation platform like SeaDroneSim2 effectively enable sim-to-real transfer for marine object detection tasks?

Key findings

• Training on only 10% of the real whale dataset augmented with synthetic data from SeaDroneSim2 improved DR50 by 15.4% and DR60 by 15.7% compared to training on real data alone.
• The FPN model with synthetic augmentation achieved a DR60 score of 0.551, a 15.7% improvement over the baseline using only 10% real data.
• The Unet model with synthetic data achieved a DR50 of 0.71, which is 3.34% higher than training on 50% of the real dataset alone.
• Synthetic data significantly reduced false negatives by including challenging cases such as partially submerged whales, which were missed by models trained only on real data.
• Performance gains were most pronounced when synthetic data was used to augment small real datasets, indicating strong data efficiency.
• The study demonstrates that synthetic data can effectively bridge the sim-to-real domain gap in marine object detection when the simulation captures realistic environmental variations.