[Paper Review] Deep Learning for Automatic Quality Grading of Mangoes: Methods and Insights

Shih-Lun Wu, Hsiao-Yen Tung+1 more|arXiv (Cornell University)|Nov 23, 2020

Smart Agriculture and AI35 references4 citations

TL;DR

This paper proposes a deep learning approach for automatic mango quality grading using convolutional neural networks (CNNs), including VGG16, ResNet, and custom convolutional autoencoder-classifiers, with transfer learning and background removal via Mask R-CNN. The best-performing model, VGG16 with preprocessed images, achieved high accuracy, and saliency maps and PCA revealed that the model focuses on visible defects like black spots, providing explainable insights into its decisions.

ABSTRACT

The quality grading of mangoes is a crucial task for mango growers as it vastly affects their profit. However, until today, this process still relies on laborious efforts of humans, who are prone to fatigue and errors. To remedy this, the paper approaches the grading task with various convolutional neural networks (CNN), a tried-and-tested deep learning technology in computer vision. The models involved include Mask R-CNN (for background removal), the numerous past winners of the ImageNet challenge, namely AlexNet, VGGs, and ResNets; and, a family of self-defined convolutional autoencoder-classifiers (ConvAE-Clfs) inspired by the claimed benefit of multi-task learning in classification tasks. Transfer learning is also adopted in this work via utilizing the ImageNet pretrained weights. Besides elaborating on the preprocessing techniques, training details, and the resulting performance, we go one step further to provide explainable insights into the model's working with the help of saliency maps and principal component analysis (PCA). These insights provide a succinct, meaningful glimpse into the intricate deep learning black box, fostering trust, and can also be presented to humans in real-world use cases for reviewing the grading results.

Motivation & Objective

• Address the labor-intensive and error-prone manual grading of mangoes, which significantly impacts growers' profitability.
• Develop an automated, accurate, and explainable system for classifying mangoes into quality grades A, B, and C using deep learning.
• Improve model performance through data preprocessing, including background removal and transfer learning using ImageNet-pretrained weights.
• Provide interpretability into the model’s decision-making process using saliency maps and principal component analysis (PCA).

Proposed method

• Employed Mask R-CNN fine-tuned on manually annotated mango boundaries to remove noisy backgrounds from 6,400 mango images.
• Utilized transfer learning by initializing models (AlexNet, VGGs, ResNets) with ImageNet-pretrained weights to improve performance on limited data.
• Designed a family of convolutional autoencoder-classifiers (ConvAE-Clfs) to leverage multi-task learning for feature regularization and transferability.
• Applied saliency maps to visualize which image regions the model focuses on during prediction, enhancing model interpretability.
• Used principal component analysis (PCA) on latent feature representations to uncover how quality grades are encoded in the model’s internal representations.
• Trained and evaluated multiple CNN architectures on the AICUP2020 Irwin mango dataset, comparing performance and interpretability across models.

Figure 1: Some sample images from AICUP2020, the dataset used in our work. Please note that the images possess high variance in background, lighting conditions, and the distance of target mangoes.

Experimental results

Research questions

• RQ1 How effective are pre-trained CNNs such as VGG16 and ResNet in classifying mango quality grades when fine-tuned on a limited, real-world dataset?
• RQ2 To what extent does background removal using Mask R-CNN improve the performance of deep learning models in mango grading?
• RQ3 Can the integration of a convolutional autoencoder with a classifier enhance model generalization and feature learning in low-data regimes?
• RQ4 What visual and feature-level patterns do deep learning models learn to distinguish between mango quality grades A, B, and C?
• RQ5 How can saliency maps and PCA provide meaningful, interpretable insights into the model’s decision-making process for real-world deployment?

Key findings

• VGG16 achieved the highest classification accuracy among all models tested, outperforming AlexNet, ResNets, and the custom ConvAE-Clfs.
• Background removal using Mask R-CNN significantly improved model performance by reducing noise and enabling the model to focus on the mango and its defects.
• Saliency maps confirmed that the model attends to visible defects such as black spots, validating that it learns relevant visual cues for grading.
• PCA analysis revealed that the first principal component strongly correlates with defect severity—highest for grade C, followed by B and A—indicating that defect load is the primary determinant of model prediction.
• Grade A mangoes were densely clustered in the latent feature space, while grade C samples were scattered, suggesting that high-quality mangoes are visually consistent, whereas low-quality ones exhibit diverse defect patterns.
• Label inconsistencies were observed in the dataset, with some mangoes showing uneven skin color or minor defects being inconsistently labeled across grades, which may explain misclassification on certain samples.

Figure 2: Background removal process. Image (a) shows how we annotate our dataset; the boundary of the mango fruit is marked with a blue polygon. As for segmentation results, Image (b) uses splash method, with which a rugged boundary of the mango is obtained. Image (c) uses the bounding box method,