[Paper Review] A comprehensive review of 3D convolutional neural network-based classification techniques of diseased and defective crops using non-UAV-based hyperspectral images

Nooshin Noshiri, Michael A. Beck+2 more|arXiv (Cornell University)|Jun 15, 2023

Spectroscopy and Chemometric AnalysesChemistry80 references3 citations

TL;DR

This paper reviews 3D convolutional neural network (3D-CNN) techniques for classifying diseased and defective crops using non-UAV-based hyperspectral imaging (HSI), emphasizing their ability to simultaneously extract spatial and spectral features for improved accuracy. It highlights challenges like data scarcity and computational complexity, and proposes solutions such as transfer learning, active learning, and spectral band selection to enhance model efficiency and performance on limited HSI datasets.

ABSTRACT

Hyperspectral imaging (HSI) is a non-destructive and contactless technology that provides valuable information about the structure and composition of an object. It can capture detailed information about the chemical and physical properties of agricultural crops. Due to its wide spectral range, compared with multispectral- or RGB-based imaging methods, HSI can be a more effective tool for monitoring crop health and productivity. With the advent of this imaging tool in agrotechnology, researchers can more accurately address issues related to the detection of diseased and defective crops in the agriculture industry. This allows to implement the most suitable and accurate farming solutions, such as irrigation and fertilization before crops enter a damaged and difficult-to-recover phase of growth in the field. While HSI provides valuable insights into the object under investigation, the limited number of HSI datasets for crop evaluation presently poses a bottleneck. Dealing with the curse of dimensionality presents another challenge due to the abundance of spectral and spatial information in each hyperspectral cube. State-of-the-art methods based on 1D- and 2D-CNNs struggle to efficiently extract spectral and spatial information. On the other hand, 3D-CNN-based models have shown significant promise in achieving better classification and detection results by leveraging spectral and spatial features simultaneously. Despite the apparent benefits of 3D-CNN-based models, their usage for classification purposes in this area of research has remained limited. This paper seeks to address this gap by reviewing 3D-CNN-based architectures and the typical deep learning pipeline, including preprocessing and visualization of results, for the classification of hyperspectral images of diseased and defective crops. Furthermore, we discuss open research areas and challenges when utilizing 3D-CNNs with HSI data.

Motivation & Objective

• To address the limited adoption of 3D-CNNs in hyperspectral image (HSI) classification for crop disease and defect detection despite their superior feature extraction capability.
• To identify key challenges in HSI-based crop classification, including data scarcity, high dimensionality, and computational complexity.
• To review deep learning pipelines—preprocessing, model architecture, and visualization—specifically tailored for 3D-CNNs in non-UAV HSI applications.
• To explore strategies such as transfer learning, active learning, and informative spectral band selection to overcome data limitations and improve model efficiency.
• To advocate for lightweight models and MLaaS integration to enable real-time, cost-effective deployment in agricultural settings.

Proposed method

• Utilizes 3D-CNN architectures that process hyperspectral cubes as 3D tensors (M×N×λ), capturing joint spatial and spectral features.
• Applies transfer learning to adapt pre-trained models on large datasets to small HSI crop datasets, improving performance with limited data.
• Employs active learning to iteratively select the most informative samples for labeling, reducing annotation burden and improving model accuracy.
• Analyzes spectral indices such as NDVI and GCI to enhance feature representation and support model interpretability.
• Explores band reduction and grouping into indices to minimize model complexity, training time, and memory footprint.
• Proposes leveraging Machine Learning as a Service (MLaaS) platforms to reduce infrastructure demands and accelerate model development for non-expert users.

Figure 1: The hyperspectral cube (adapted from Tarabalka et al. ( 2010 ) with modification). It is a three-dimensional array where each pixel represents a spectrum containing a range of wavelengths. This spectrum can act as a fingerprint and provides information about biophysical and biochemical cha

Experimental results

Research questions

• RQ1 How can 3D-CNNs effectively extract both spatial and spectral features from non-UAV-based hyperspectral images for crop disease and defect classification?
• RQ2 What are the primary challenges in applying 3D-CNNs to hyperspectral crop imaging, particularly regarding data scarcity and computational cost?
• RQ3 To what extent can transfer learning and active learning improve the performance of 3D-CNN models trained on limited HSI datasets?
• RQ4 Which spectral bands are most informative for detecting crop diseases and defects, and how can they be selected to build lightweight, efficient models?
• RQ5 How can MLaaS platforms and model optimization techniques enable real-time, field-deployable 3D-CNN systems in precision agriculture?

Key findings

• 3D-CNNs outperform 1D and 2D-CNNs in classifying hyperspectral crop images by jointly modeling spatial and spectral features, leading to higher accuracy.
• Transfer learning significantly improves model performance on small HSI datasets by leveraging pre-trained weights, reducing overfitting and training time.
• Active learning reduces the number of required labeled samples by selecting the most informative instances, enhancing learning efficiency.
• Spectral band selection and grouping into indices can reduce model complexity, decrease training time, and maintain or improve classification accuracy.
• MLaaS platforms can lower barriers to entry for agricultural practitioners by providing access to pre-built 3D-CNN models and scalable computing infrastructure.
• Despite their promise, 3D-CNNs remain underutilized in HSI-based crop classification due to data scarcity and computational demands, indicating a need for further research and tooling.

Figure 2: A basic conceptual CNN architecture. A CNN consists of multiple layers, including convolution, detector, and pooling layers, where each layer serves as the input for the subsequent layer, enabling the extraction of low-dimensional features, learning of non-linear representations, and dimen