[Paper Review] An Application of Deep Learning for Sweet Cherry Phenotyping using YOLO Object Detection

Ritayu Nagpal, Sam Long+5 more|arXiv (Cornell University)|Feb 13, 2023

Smart Agriculture and AIAgricultural and Biological Sciences3 citations

TL;DR

This paper proposes a YOLOv3-based deep learning system for automated sweet cherry phenotyping, enabling real-time detection, counting, size estimation, and color classification from images. The model achieves 99% accuracy in object detection and 90% Intersection over Union (IoU) for localization, significantly outperforming manual methods in speed and consistency while reducing human error and bias.

ABSTRACT

Tree fruit breeding is a long-term activity involving repeated measurements of various fruit quality traits on a large number of samples. These traits are traditionally measured by manually counting the fruits, weighing to indirectly measure the fruit size, and fruit colour is classified subjectively into different color categories using visual comparison to colour charts. These processes are slow, expensive and subject to evaluators' bias and fatigue. Recent advancements in deep learning can help automate this process. A method was developed to automatically count the number of sweet cherry fruits in a camera's field of view in real time using YOLOv3. A system capable of analyzing the image data for other traits such as size and color was also developed using Python. The YOLO model obtained close to 99% accuracy in object detection and counting of cherries and 90% on the Intersection over Union metric for object localization when extracting size and colour information. The model surpasses human performance and offers a significant improvement compared to manual counting.

Motivation & Objective

• To automate the phenotyping of sweet cherries in breeding programs to overcome time-consuming, error-prone manual methods.
• To develop a real-time, accurate, and consistent system for counting, sizing, and color classification of cherries using deep learning.
• To reduce human bias and fatigue in fruit trait assessment by replacing subjective visual comparisons with objective, data-driven analysis.
• To create an integrated desktop application using Python, OpenCV, and Tkinter for end-to-end image analysis and data export.
• To demonstrate that deep learning models can surpass human performance in speed and accuracy for cherry phenotyping tasks.

Proposed method

• Employed YOLOv3 and YOLOv3 EfficientNet for object detection and localization in cherry images.
• Trained models using transfer learning on a custom-labeled dataset of sweet cherry images captured in controlled environments.
• Used bounding box regression and confidence scores to estimate cherry size from image dimensions.
• Applied color classification using mean RGB values from detected bounding boxes, compared against CTIFL color standards.
• Integrated the model with OpenCV and Tkinter to build a desktop application for real-time analysis and data export to spreadsheets.
• Optimized inference speed and accuracy through hyperparameter tuning of confidence and non-max suppression thresholds.

Experimental results

Research questions

• RQ1 Can YOLO-based object detection achieve high accuracy in counting and localizing sweet cherries in real-world agricultural images?
• RQ2 How does the performance of YOLOv3 compare to traditional manual phenotyping methods in terms of speed, accuracy, and consistency?
• RQ3 To what extent can deep learning models accurately estimate cherry size and color from image data, and how do they compare to human evaluations?
• RQ4 Can the automated system detect and correct errors present in previously collected manual data?
• RQ5 What is the impact of image preprocessing, lighting variation, and white balance on the reliability of color classification in phenotyping applications?

Key findings

• The YOLOv3 model achieved 99% accuracy in detecting and counting sweet cherries, significantly outperforming manual counting in speed and consistency.
• The model achieved a 90% Intersection over Union (IoU) score, indicating high precision in object localization for size and color estimation.
• Correlation between YOLO-predicted cherry size and traditional weight-based measurements was 0.875 (p < 0.001), explaining 76.5% of variance.
• Color classification accuracy reached 72%, which experts deemed acceptable given human subjectivity and lighting variability.
• The system reduced processing time by 95% compared to manual methods, enabling high-throughput phenotyping.
• The model successfully identified and corrected manual errors in prior year’s image datasets, demonstrating robustness and reliability.