[Paper Review] AgriNeRF: Neural Radiance Fields for Agriculture in Challenging Lighting Conditions

Samarth Chopra, Fernando Cladera+2 more|arXiv (Cornell University)|Sep 23, 2024

Remote Sensing in Agriculture4 citations

TL;DR

AgriNeRF proposes a multi-modal Neural Radiance Field framework fusing RGB, event, and thermal cameras to enable robust 3D scene reconstruction and fruit detection under challenging lighting conditions in agriculture. It achieves +2.06 dB PSNR and +8.3% SSIM in RGB reconstruction, and a +44.8% mAP50 gain in fruit detection via cross-spectral fusion, significantly outperforming single-modality baselines in low-light and variable lighting scenarios.

ABSTRACT

Neural Radiance Fields (NeRFs) have shown significant promise in 3D scene reconstruction and novel view synthesis. In agricultural settings, NeRFs can serve as digital twins, providing critical information about fruit detection for yield estimation and other important metrics for farmers. However, traditional NeRFs are not robust to challenging lighting conditions, such as low-light, extreme bright light and varying lighting. To address these issues, this work leverages three different sensors: an RGB camera, an event camera and a thermal camera. Our RGB scene reconstruction shows an improvement in PSNR and SSIM by +2.06 dB and +8.3% respectively. Our cross-spectral scene reconstruction enhances downstream fruit detection by +43.0% in mAP50 and +61.1% increase in mAP50-95. The integration of additional sensors leads to a more robust and informative NeRF. We demonstrate that our multi-modal system yields high quality photo-realistic reconstructions under various tree canopy covers and at different times of the day. This work results in the development of a resilient NeRF, capable of performing well in visibly degraded scenarios, as well as a learnt cross-spectral representation, that is used for automated fruit detection.

Motivation & Objective

• To address the poor performance of traditional NeRFs in agricultural settings under challenging lighting conditions such as low-light, high dynamic range, and variable illumination.
• To improve 3D scene reconstruction fidelity in real-world orchards and gardens where lighting varies significantly throughout the day.
• To develop a cross-spectral NeRF representation that enhances downstream fruit detection by leveraging complementary information from visible and infrared bands.
• To create a novel, real-world multi-modal dataset combining RGB, event, thermal, LiDDR, and IMU data for agricultural NeRF research.
• To demonstrate that multi-modal fusion leads to more resilient and accurate digital twins for precision agriculture applications.

Proposed method

• Integrates three sensors—RGB, event, and thermal cameras—into a unified sensor suite for capturing multi-spectral scene data under diverse lighting conditions.
• Proposes a regularization loss that maximizes feature correlation between RGB, event, and thermal frames to improve RGB reconstruction fidelity.
• Trains a joint cross-spectral NeRF that fuses features from visible (RGB) and infrared (thermal) bands to generate a robust, multi-spectral 3D scene representation.
• Uses event camera data to enhance temporal resolution and mitigate motion blur, improving reconstruction quality in dynamic or poorly lit scenes.
• Employs a multi-view NeRF architecture with positional encoding and volume rendering, adapted to handle cross-spectral feature fusion.
• Leverages a self-built sensor suite with synchronized data acquisition across RGB, thermal, event, LiDAR, and IMU sensors for real-world data collection.

Experimental results

Research questions

• RQ1 Can the fusion of RGB, event, and thermal cameras significantly improve NeRF reconstruction quality in agricultural environments with challenging lighting conditions?
• RQ2 How does cross-spectral NeRF reconstruction enhance downstream fruit detection performance compared to single-modality RGB NeRFs?
• RQ3 What is the contribution of each individual modality (RGB, event, thermal) to the final fruit detection performance in low-light and high-dynamic-range scenarios?
• RQ4 To what extent does the proposed regularization loss improve feature alignment and reconstruction fidelity across multi-modal inputs?
• RQ5 How does the performance of AgriNeRF vary across different lighting conditions—e.g., morning, midday, evening, and shadowed areas—compared to state-of-the-art NeRF methods?

Key findings

• AgriNeRF improves RGB scene reconstruction by +2.06 dB in PSNR and +8.3% in SSIM compared to prior NeRF methods, demonstrating superior fidelity under variable lighting.
• The cross-spectral NeRF achieves a +44.8% increase in mAP50 and a +55.7% increase in mAP50-95 for fruit detection compared to the second-best method, especially excelling in low-light conditions.
• In low-light sequences such as the garden tomatoes and eggplants datasets, the cross-spectral reconstruction detects fruits not visible in RGB-only reconstructions, achieving up to 3× higher mAP50 than baseline methods.
• The ablation study confirms that combining all three modalities (RGB, event, thermal) yields the highest fruit detection performance, with a +65.3% increase in mAP50 and a +608% boost in recall compared to RGB-only.
• Thermal imaging dominates in low-light sequences due to the thermal signature of fruits, while event cameras contribute to motion blur reduction and dynamic range extension.
• Despite strong performance in most conditions, the system struggles in complete darkness due to lack of visible light or thermal emission, highlighting a limitation for night-time applications.