[Paper Review] Dataset and Performance Comparison of Deep Learning Architectures for Plum Detection and Robotic Harvesting

Jasper Brown, Salah Sukkarieh|arXiv (Cornell University)|May 9, 2021

Smart Agriculture and AI29 references4 citations

TL;DR

This paper introduces two new RGB-D datasets collected during real-world robotic plum harvesting under day and night conditions, benchmarking four deep learning object detectors (RetinaNet, Faster R-CNN, YOLOv3, CenterNet). It finds transfer learning is essential for small datasets, night detection is more accurate than day for some models, and RGBD fusion offers only marginal gains, with data augmentation proving more effective than depth integration.

ABSTRACT

Many automated operations in agriculture, such as weeding and plant counting, require robust and accurate object detectors. Robotic fruit harvesting is one of these, and is an important technology to address the increasing labour shortages and uncertainty suffered by tree crop growers. An eye-in-hand sensing setup is commonly used in harvesting systems and provides benefits to sensing accuracy and flexibility. However, as the hand and camera move from viewing the entire trellis to picking a specific fruit, large changes in lighting, colour, obscuration and exposure occur. Object detection algorithms used in harvesting should be robust to these challenges, but few datasets for assessing this currently exist. In this work, two new datasets are gathered during day and night operation of an actual robotic plum harvesting system. A range of current generation deep learning object detectors are benchmarked against these. Additionally, two methods for fusing depth and image information are tested for their impact on detector performance. Significant differences between day and night accuracy of different detectors is found, transfer learning is identified as essential in all cases, and depth information fusion is assessed as only marginally effective. The dataset and benchmark models are made available online.

Motivation & Objective

• To address the lack of realistic, application-specific datasets for evaluating deep learning object detectors in robotic fruit harvesting under variable lighting and occlusion.
• To benchmark state-of-the-art deep learning object detectors (RetinaNet, Faster R-CNN, YOLOv3, CenterNet) on real-world day and night harvesting conditions.
• To evaluate the impact of fusing depth and image data using early and late fusion strategies on detection performance.
• To assess the role of transfer learning and data augmentation in improving detector accuracy on small agricultural datasets.
• To provide publicly available datasets and trained models to support future research in agricultural robotics and fruit detection.

Proposed method

• Collected 700 annotated RGB-D images (350 day, 350 night) from an eye-in-hand robotic plum harvesting system using an Intel Realsense D435 camera.
• Trained and evaluated four deep learning object detection architectures—RetinaNet, Faster R-CNN, YOLOv3, and CenterNet—on the day and night datasets.
• Implemented two RGBD fusion strategies: early fusion (concatenating RGB and depth as 4-channel input) and late fusion (concatenating features from parallel RGB and depth backbones).
• Applied transfer learning using ImageNet-pretrained weights and data augmentation (random cropping, color jittering) to improve performance on small datasets.
• Used mean average precision (mAP) at 0.5 IOU threshold to evaluate detection performance across all models and configurations.
• Conducted ablation studies to isolate the effects of backbone architecture, transfer learning, and fusion strategy on detection accuracy.

Experimental results

Research questions

• RQ1 How do state-of-the-art deep learning object detectors perform on real-world robotic plum harvesting data collected under day and night conditions?
• RQ2 What is the impact of transfer learning on detector performance when training on small agricultural datasets?
• RQ3 Does early or late fusion of RGB and depth data improve object detection accuracy in robotic harvesting scenarios?
• RQ4 Why do detection performance differences between day and night datasets vary significantly across models, especially given that night conditions have fewer visible but obscured fruits?
• RQ5 How does the performance of models on this custom dataset compare to their reported performance on the COCO benchmark?

Key findings

• Transfer learning was essential for all models, significantly improving performance on the small dataset, with greater impact than architecture choice.
• Faster R-CNN achieved the highest average precision on the night dataset (mAP 0.78), outperforming more modern and faster networks like YOLOv3 and RetinaNet.
• RetinaNet achieved the highest mAP on the day dataset (0.82) when combined with data augmentation and transfer learning using a ResNet-101 backbone.
• Late fusion of RGB and depth features marginally improved performance over RGB-only baseline, but early fusion degraded performance.
• Data augmentation provided a larger performance boost than depth fusion, suggesting that expanding the RGB dataset would be more effective than adding depth information.
• CenterNet performed poorly on both datasets despite high COCO mAP, indicating that COCO performance is not a reliable predictor for agricultural detection tasks.