[Paper Review] AeroGen: Enhancing Remote Sensing Object Detection with Diffusion-Driven Data Generation

Datao Tang, Xiangyong Cao+7 more|arXiv (Cornell University)|Nov 23, 2024

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

AeroGen proposes a layout-controllable diffusion model for remote sensing object detection (RSIOD) that generates high-quality synthetic images with both horizontal and rotated bounding box conditions. By integrating a diversity-conditioned generator and a dual filtering mechanism, the method enhances data diversity and quality, improving mAP by 3.7%, 4.3%, and 2.43% on DIOR, DIOR-R, and HRSC datasets respectively, with significant gains in rare object classes.

ABSTRACT

Remote sensing image object detection (RSIOD) aims to identify and locate specific objects within satellite or aerial imagery. However, there is a scarcity of labeled data in current RSIOD datasets, which significantly limits the performance of current detection algorithms. Although existing techniques, e.g., data augmentation and semi-supervised learning, can mitigate this scarcity issue to some extent, they are heavily dependent on high-quality labeled data and perform worse in rare object classes. To address this issue, this paper proposes a layout-controllable diffusion generative model (i.e. AeroGen) tailored for RSIOD. To our knowledge, AeroGen is the first model to simultaneously support horizontal and rotated bounding box condition generation, thus enabling the generation of high-quality synthetic images that meet specific layout and object category requirements. Additionally, we propose an end-to-end data augmentation framework that integrates a diversity-conditioned generator and a filtering mechanism to enhance both the diversity and quality of generated data. Experimental results demonstrate that the synthetic data produced by our method are of high quality and diversity. Furthermore, the synthetic RSIOD data can significantly improve the detection performance of existing RSIOD models, i.e., the mAP metrics on DIOR, DIOR-R, and HRSC datasets are improved by 3.7%, 4.3%, and 2.43%, respectively. The code is available at https://github.com/Sonettoo/AeroGen.

Motivation & Objective

• Address the scarcity of labeled remote sensing image data, which limits the performance of object detection models.
• Overcome the limitations of traditional data augmentation and semi-supervised learning in rare object detection by generating diverse, high-quality synthetic data.
• Develop a generative model capable of conditional image synthesis based on both horizontal and rotated bounding box layouts for remote sensing images.
• Design an end-to-end data augmentation pipeline that improves both diversity and quality of synthetic training data for downstream detection tasks.
• Demonstrate that synthetic data from the proposed method significantly improves detection performance, especially for rare object categories.

Proposed method

• Propose AeroGen, a diffusion generative model conditioned on layout priors using both horizontal and rotated bounding boxes to guide image generation.
• Integrate Layout Mask Attention (LMA) and Dual Cross Attention (DCA) modules to enhance global semantic understanding and object localization in generated images.
• Design a diversity-conditioned generator that produces varied layout conditions to increase the diversity of synthetic data.
• Implement a double filtering mechanism: one for filtering low-quality layout conditions and another for filtering low-quality generated images based on semantic and layout consistency.
• Construct an end-to-end data augmentation pipeline that directly synthesizes RSIOD data without post-paste operations, improving training efficiency and data fidelity.
• Use FID, CAS, and YOLO score as evaluation metrics to assess image quality and generation fidelity during training and ablation studies.

Figure 1 : AeroGen’s overall architecture. (a) The layout embedding module combines bounding box coordinates with vectorized semantic information using Fourier and MLP layers. This encodes layout information to facilitate control, with the prompt description processed by a CLIP text encoder for glob

Experimental results

Research questions

• RQ1 Can a diffusion model be effectively adapted to generate high-quality remote sensing images with precise layout control for object detection?
• RQ2 Does the integration of both horizontal and rotated bounding box conditions improve the realism and utility of synthetic remote sensing data?
• RQ3 To what extent does the proposed end-to-end data augmentation pipeline enhance detection performance compared to traditional augmentation methods?
• RQ4 How effective is the dual filtering mechanism in maintaining semantic and layout consistency in generated synthetic data?
• RQ5 Can synthetic data generated by AeroGen significantly improve detection performance on rare object categories in remote sensing datasets?

Key findings

• The proposed AeroGen model achieves state-of-the-art performance in layout-controllable generation for remote sensing images, supporting both horizontal and rotated bounding box conditions.
• On the DIOR dataset, synthetic data improved mAP by 3.7% and mAP50 by 4.3% when using 50k generated samples.
• On the DIOR-R dataset, the mAP increased by 4.3% and mAP50 by 4.8% with 50k synthetic samples, with notable gains in rare classes such as GF (+17.8%), DAM (+14.7%), and APO (+12.6%).
• On the HRSC dataset, the mAP improved by 2.43% using 50k synthetic samples, demonstrating consistent performance gains across diverse datasets.
• The ablation study confirms that both Layout Mask Attention (LMA) and Dual Cross Attention (DCA) significantly improve image quality, with FID dropping from 82.11 to 38.57 when both are used.
• The end-to-end pipeline with filtering and diversity conditioning outperforms baseline methods: AeroGen alone achieved 41.69 mAP on DIOR-R, surpassing Copy-Paste + Flip (38.75 mAP) and Flip (37.39 mAP).

Figure 2 : Overview of the pipeline based on AeroGen. By fitting the conditional distribution using a diffusion model, we expand a diverse set of layout conditions and combine them with AeroGen to generate synthetic data. Additionally, we introduce two filters to eliminate low-quality synthetic cond