[Paper Review] Conditional Random Field and Deep Feature Learning for Hyperspectral Image Segmentation

Fahim Irfan Alam, Jun Zhou+4 more|arXiv (Cornell University)|Nov 13, 2017

Remote-Sensing Image Classification27 references3 citations

TL;DR

This paper proposes a novel deep learning framework that integrates 3D convolutional neural networks (3D-CNN) with deep conditional random fields (CRF) for hyperspectral image segmentation. By leveraging spectral-spatial features through 3D-CNNs and refining predictions via CRF with CNN-based potentials and a deconvolution block, the method achieves state-of-the-art performance on benchmark and a new dataset, with inference times under 1.1 minutes per image.

ABSTRACT

Image segmentation is considered to be one of the critical tasks in hyperspectral remote sensing image processing. Recently, convolutional neural network (CNN) has established itself as a powerful model in segmentation and classification by demonstrating excellent performances. The use of a graphical model such as a conditional random field (CRF) contributes further in capturing contextual information and thus improving the segmentation performance. In this paper, we propose a method to segment hyperspectral images by considering both spectral and spatial information via a combined framework consisting of CNN and CRF. We use multiple spectral cubes to learn deep features using CNN, and then formulate deep CRF with CNN-based unary and pairwise potential functions to effectively extract the semantic correlations between patches consisting of three-dimensional data cubes. Effective piecewise training is applied in order to avoid the computationally expensive iterative CRF inference. Furthermore, we introduce a deep deconvolution network that improves the segmentation masks. We also introduce a new dataset and experimented our proposed method on it along with several widely adopted benchmark datasets to evaluate the effectiveness of our method. By comparing our results with those from several state-of-the-art models, we show the promising potential of our method.

Motivation & Objective

• To address the challenge of hyperspectral image segmentation by combining spectral and spatial information effectively.
• To improve segmentation accuracy by integrating conditional random fields (CRF) with deep neural networks to model contextual dependencies.
• To reduce overfitting in deep learning models through data augmentation and expanded training samples.
• To enhance initial CNN predictions using a deconvolution block within the CRF framework for refined segmentation masks.
• To evaluate the method on both benchmark datasets and a newly introduced dataset for comprehensive performance assessment.

Proposed method

• Utilizes 3D-CNNs to extract deep spectral-spatial features from multiple spectral cubes, capturing 3D data correlations across bands.
• Constructs a deep CRF where unary and pairwise potentials are derived from CNN features, enabling end-to-end learning of contextual relationships.
• Incorporates a deconvolution block into the CRF pairwise potential computation to refine segmentation masks and improve spatial consistency.
• Employs data augmentation techniques to increase training sample size and mitigate overfitting in the CNN training process.
• Applies mean field inference with efficient approximations to reduce CRF computation complexity to linear time in the number of labels and edges.
• Trains the joint CNN-CRF model in an end-to-end fashion, optimizing both feature learning and CRF parameter estimation.

Experimental results

Research questions

• RQ1 Can a joint 3D-CNN and deep CRF framework improve hyperspectral image segmentation beyond standalone CNNs or CRFs?
• RQ2 How effective is the integration of CNN-based unary and pairwise potentials in a deep CRF for modeling spatial-spectral dependencies?
• RQ3 To what extent does the inclusion of a deconvolution block in the CRF enhance segmentation mask quality and spatial coherence?
• RQ4 How does the proposed method compare in performance and inference efficiency to state-of-the-art models on both public and newly introduced datasets?
• RQ5 Can data augmentation significantly improve generalization and reduce overfitting in hyperspectral image segmentation with limited training samples?

Key findings

• The proposed method achieved the highest overall accuracy on the Indian Pines dataset (91.25%) and Pavia University (92.15%), outperforming existing state-of-the-art models.
• On the newly introduced Griffith-USGS dataset, the method achieved 85.97% overall accuracy, demonstrating strong generalization across diverse datasets.
• The method achieved inference times of approximately 1.08 minutes on the Griffith-USGS dataset, comparable to other deep learning baselines and suitable for real-time applications.
• The integration of the deconvolution block into the CRF pairwise potential significantly improved the smoothness and accuracy of segmentation masks.
• Data augmentation effectively reduced overfitting and improved model generalization, especially in low-data regimes.
• The deep CRF framework with CNN-derived potentials demonstrated superior performance in capturing long-range contextual dependencies compared to traditional CRF or standalone CNN approaches.