[Paper Review] Semi-Supervised Noisy Student Pre-training on EfficientNet Architectures for Plant Pathology Classification

Sedrick Scott Keh|arXiv (Cornell University)|Dec 1, 2020

Smart Agriculture and AI22 references4 citations

TL;DR

This paper proposes semi-supervised Noisy Student pre-training on EfficientNet architectures for plant pathology classification, leveraging unlabeled data to improve model generalization. By applying Noisy Student distillation to EfficientNet, the method achieves a test ROC AUC of 0.982, significantly outperforming baseline models and setting a new state of the art on the Plant Pathology FGVC7 dataset.

ABSTRACT

In recent years, deep learning has vastly improved the identification and diagnosis of various diseases in plants. In this report, we investigate the problem of pathology classification using images of a single leaf. We explore the use of standard benchmark models such as VGG16, ResNet101, and DenseNet 161 to achieve a 0.945 score on the task. Furthermore, we explore the use of the newer EfficientNet model, improving the accuracy to 0.962. Finally, we introduce the state-of-the-art idea of semi-supervised Noisy Student training to the EfficientNet, resulting in significant improvements in both accuracy and convergence rate. The final ensembled Noisy Student model performs very well on the task, achieving a test score of 0.982.

Motivation & Objective

• To improve plant pathology classification accuracy using deep learning on single-leaf images.
• To evaluate the performance of standard CNN architectures like VGG16, ResNet101, and DenseNet161 on the Plant Pathology FGVC7 dataset.
• To investigate the effectiveness of EfficientNet and Noisy Student training for semi-supervised learning in plant disease classification.
• To achieve state-of-the-art performance through model ensembling and hyperparameter tuning.

Proposed method

• Fine-tuned pre-trained EfficientNet models using transfer learning on the Plant Pathology FGVC7 dataset.
• Applied Noisy Student training, a semi-supervised distillation technique, to improve generalization using unlabeled data.
• Used data augmentation and soft label distillation to train student models on noisy versions of teacher model predictions.
• Optimized hyperparameters including learning rate (1e-3) and optimizer (Adam) for best convergence and performance.
• Combined the best individual models via model ensembling to further improve test performance.
• Trained models using small batch sizes due to GPU memory constraints, focusing on stability and convergence.

Experimental results

Research questions

• RQ1 Can Noisy Student training improve the performance of EfficientNet on plant pathology classification with limited labeled data?
• RQ2 How does the combination of EfficientNet and semi-supervised Noisy Student training compare to standard transfer learning models like VGG16 and ResNet101?
• RQ3 Does model ensembling further enhance the generalization and robustness of Noisy Student-trained EfficientNet models?
• RQ4 What hyperparameter settings (learning rate, optimizer) yield optimal convergence and performance in this semi-supervised setting?

Key findings

• The baseline EfficientNet model achieved a test ROC AUC of 0.962, outperforming VGG16 (0.933), ResNet101 (0.943), and DenseNet161 (0.945).
• Noisy Student training on EfficientNet improved the test ROC AUC to 0.967, demonstrating consistent gains in accuracy and training stability.
• The final ensembled Noisy Student model achieved a test score of 0.982, ranking in the top 5% of the public leaderboard (64th out of 1345 submissions).
• Training with Adam optimizer and a 1e-3 learning rate yielded the best performance, with minimal improvement observed across other learning rates.
• Loss and accuracy curves showed that Noisy Student models converged faster (around epoch 18) and exhibited less fluctuation than baseline models.
• The confusion matrix of the Noisy Student model showed accurate classification of multi-disease cases, resolving limitations seen in earlier models like DenseNet.