[Paper Review] An End-to-End Two-Phase Deep Learning-Based workflow to Segment Man-made Objects Around Reservoirs

Nayereh Hamidishad, Roberto Marcondes Cesar Junior|arXiv (Cornell University)|Feb 7, 2023

Flood Risk Assessment and Management4 citations

TL;DR

This paper proposes an end-to-end two-phase deep learning workflow for segmenting man-made objects around reservoirs using Google Earth imagery. First, a U-Net or SegNet model segments reservoirs, followed by post-processing to remove floating vegetation; then, a region of interest around the reservoir is identified, and man-made objects are segmented using U-Net, FPN, LinkNet, or PSPNet, achieving 96.53% F1-score in phase one and 90.32% in phase two.

ABSTRACT

Reservoirs are fundamental infrastructures for the management of water resources. Constructions around them can negatively impact their quality. Such unauthorized constructions can be monitored by land cover mapping (LCM) remote sensing (RS) images. In this paper, we develop a new approach based on DL and image processing techniques for man-made object segmentation around the reservoirs. In order to segment man-made objects around the reservoirs in an end-to-end procedure, segmenting reservoirs and identifying the region of interest (RoI) around them are essential. In the proposed two-phase workflow, the reservoir is initially segmented using a DL model. A post-processing stage is proposed to remove errors such as floating vegetation. Next, the RoI around the reservoir (RoIaR) is identified using the proposed image processing techniques. Finally, the man-made objects in the RoIaR are segmented using a DL architecture. We trained the proposed workflow using collected Google Earth (GE) images of eight reservoirs in Brazil over two different years. The U-Net-based and SegNet-based architectures are trained to segment the reservoirs. To segment man-made objects in the RoIaR, we trained and evaluated four possible architectures, U-Net, FPN, LinkNet, and PSPNet. Although the collected data has a high diversity (for example, they belong to different states, seasons, resolutions, etc.), we achieved good performances in both phases. Furthermore, applying the proposed post-processing to the output of reservoir segmentation improves the precision in all studied reservoirs except two cases. We validated the prepared workflow with a reservoir dataset outside the training reservoirs. The results show high generalization ability of the prepared workflow.

Motivation & Objective

• To develop an end-to-end deep learning workflow for segmenting man-made objects around reservoirs using remote sensing imagery.
• To address the challenge of detecting unauthorized constructions near reservoirs through automated land cover mapping using high-resolution satellite images.
• To improve segmentation accuracy by introducing a two-phase approach with post-processing and region-of-interest (RoIaR) localization.
• To validate the model’s generalization on unseen reservoirs outside the training data, ensuring robustness across diverse geographical and seasonal conditions.

Proposed method

• Use U-Net and SegNet architectures to perform initial semantic segmentation of reservoirs in high-resolution Google Earth images.
• Apply a post-processing stage to correct segmentation errors, particularly removing floating vegetation, using morphological operations and spectral filtering.
• Define a region of interest around the reservoir (RoIaR) using image processing techniques such as dilation and contour detection to focus on relevant areas.
• Train and evaluate four deep learning architectures—U-Net, FPN, LinkNet, and PSPNet—for man-made object segmentation within the RoIaR.
• Optimize training using a combination of Dice and Focal loss functions and apply over-sampling to handle class imbalance in rural man-made object detection.
• Evaluate model performance using F1-score on both training and external validation datasets, including a reservoir not in the training set.

Figure 1: Overview of the proposed analysis workflow.

Experimental results

Research questions

• RQ1 Can a two-phase deep learning workflow achieve high-accuracy segmentation of man-made objects around reservoirs in diverse geographical and seasonal conditions?
• RQ2 How effective is post-processing in improving reservoir segmentation precision, especially in removing false positives like floating vegetation?
• RQ3 To what extent does RoIaR localization enhance the performance of man-made object segmentation compared to full-image segmentation?
• RQ4 How do different loss functions and data augmentation strategies affect model performance on imbalanced rural man-made object datasets?
• RQ5 Does the proposed workflow generalize well to unseen reservoirs with different spatial resolutions, atmospheric conditions, and land cover types?

Key findings

• The phase-1 reservoir segmentation achieved a highest F1-score of 96.53% on the test set, demonstrating high accuracy in identifying water bodies.
• Post-processing improved precision in all but two reservoirs, with the F1-score increasing to 94.68% on the external validation set after post-processing.
• The phase-2 man-made object segmentation achieved a highest F1-score of 90.32% using the PSPNet architecture with the VGG-16 backbone.
• Combining Dice and Focal loss functions significantly improved performance on imbalanced data, outperforming Focal loss alone.
• Over-sampling further enhanced segmentation results, particularly for rare rural man-made object classes.
• The workflow generalized well to an unseen reservoir (Barra), achieving 92.54% F1-score in phase-1 and 88.11% in phase-2, confirming robustness and transferability.

Figure 2: Visualization of the studied reservoirs locations.