[Paper Review] deepNIR: Datasets for Generating Synthetic NIR Images and Improved Fruit Detection System Using Deep Learning Techniques

Inkyu Sa, Jong Yoon Lim+2 more|arXiv (Cornell University)|Jun 22, 2022

Smart Agriculture and AI52 references28 citations

TL;DR

This paper introduces deepNIR, a publicly available dataset of 161,500 paired RGB-NIR images from commercial farms and reprocessed public datasets (nirscene, SEN12MS), enabling high-fidelity synthetic near-infrared (NIR) image generation via conditional GANs. The method achieves a Frechet Inception Distance (FID) of 11.36 on the nirscene subset and improves fruit detection with YOLOv5, reporting mAP[0.5:0.95] scores up to 0.812 across 11 fruit classes.

ABSTRACT

This paper presents datasets utilised for synthetic near-infrared (NIR) image generation and bounding-box level fruit detection systems. A high-quality dataset is one of the essential building blocks that can lead to success in model generalisation and the deployment of data-driven deep neural networks. In particular, synthetic data generation tasks often require more training samples than other supervised approaches. Therefore, in this paper, we share the NIR+RGB datasets that are re-processed from two public datasets (i.e., nirscene and SEN12MS), expanded our previous study, deepFruits, and our novel NIR+RGB sweet pepper (capsicum) dataset. We oversampled from the original nirscene dataset at 10, 100, 200, and 400 ratios that yielded a total of 127 k pairs of images. From the SEN12MS satellite multispectral dataset, we selected Summer (45 k) and All seasons (180k) subsets and applied a simple yet important conversion: digital number (DN) to pixel value conversion followed by image standardisation. Our sweet pepper dataset consists of 1615 pairs of NIR+RGB images that were collected from commercial farms. We quantitatively and qualitatively demonstrate that these NIR+RGB datasets are sufficient to be used for synthetic NIR image generation. We achieved Frechet inception distances (FIDs) of 11.36, 26.53, and 40.15 for nirscene1, SEN12MS, and sweet pepper datasets, respectively. In addition, we release manual annotations of 11 fruit bounding boxes that can be exported in various formats using cloud service. Four newly added fruits (blueberry, cherry, kiwi and wheat) compound 11 novel bounding box datasets on top of our previous work presented in the deepFruits project (apple, avocado, capsicum, mango, orange, rockmelon and strawberry). The total number of bounding box instances of the dataset is 162 k and it is ready to use from a cloud service. For the evaluation of the dataset, Yolov5 single stage detector is exploited and reported impressive mean-average-precision, mAP[0.5:0.95] results of min:0.49, max:0.812. We hope these datasets are useful and serve as a baseline for future studies.

Motivation & Objective

• To address the scarcity of high-quality, publicly available RGB-NIR paired datasets for training synthetic NIR image generation models in agriculture.
• To improve fruit detection performance by leveraging synthetic NIR images as auxiliary input in deep learning pipelines.
• To expand the deepFruits dataset by adding four new fruit classes (blueberry, cherry, kiwi, wheat) with precise bounding box annotations.
• To provide a reproducible, open benchmark for synthetic image generation and object detection in agricultural computer vision.
• To enable domain-specific pretraining for agricultural models using a curated, in-domain dataset instead of generic datasets like COCO or ImageNet.

Proposed method

• Reprocessed and expanded two public datasets (nirscene and SEN12MS) through oversampling (10x to 400x) and image standardization (μ=0, σ=1) to increase training data scale.
• Collected a novel, high-quality sweet pepper (capsicum) dataset of 1,615 RGB-NIR image pairs from commercial farms under consistent lighting and radiometric calibration.
• Trained a conditional generative adversarial network (cGAN) to learn the mapping from RGB to synthetic NIR images using pixel-wise and perceptual loss functions.
• Applied a Frechet Inception Distance (FID) metric to quantitatively evaluate the realism of generated synthetic NIR images.
• Fine-tuned YOLOv5 for single-stage object detection using both RGB and synthetic NIR inputs, evaluating performance with mAP[0.5:0.95] and mAP[0.5].
• Provided cloud-hosted, exportable bounding box annotations in multiple formats (e.g., COCO, YOLO) for immediate use in downstream tasks.

Experimental results

Research questions

• RQ1 Can synthetic NIR images generated from RGB inputs using deep learning achieve high perceptual and statistical fidelity suitable for agricultural applications?
• RQ2 To what extent does incorporating synthetic NIR images improve the performance of fruit detection models in real-world agricultural settings?
• RQ3 How does the scale and quality of the training dataset affect the generalization and performance of synthetic image generation models?
• RQ4 What is the impact of domain-specific pretraining using a curated agricultural dataset compared to generic pretraining on ImageNet or COCO?
• RQ5 How do different fruit types respond to synthetic NIR augmentation in terms of mAP improvement or degradation?

Key findings

• The deepNIR dataset achieved a Frechet Inception Distance (FID) of 11.36 on the nirscene1 subset, 26.53 on SEN12MS, and 40.15 on the sweet pepper dataset, indicating high realism in synthetic NIR generation.
• The YOLOv5 detector achieved a maximum mAP[0.5:0.95] of 0.812 on the expanded deepFruits dataset, demonstrating strong detection performance across 11 fruit classes.
• For apple detection, synthetic NIR input improved mAP[0.5:0.95] by 0.005, indicating a positive impact in some cases.
• Cherry and kiwi detection performance decreased by 0.03 and 0.09 mAP[0.5:0.95] respectively with synthetic NIR, suggesting domain shift or distribution mismatch issues.
• Capsicum detection showed a slight mAP[0.5:0.95] drop of 0.01 but a 0.01 increase in mAP[0.5], indicating synthetic NIR may help detect partially occluded or shadowed fruits.
• The study demonstrates that synthetic NIR generation is effective only when test and training distributions are similar, highlighting the challenge of generalization across diverse real-world conditions.