[Paper Review] Exploring Accurate 3D Phenotyping in Greenhouse through Neural Radiance Fields

unhong Zhao, Ying Wei+5 more|arXiv (Cornell University)|Mar 24, 2024

Greenhouse Technology and Climate Control5 citations

TL;DR

The paper develops a NeRF-based method for in-situ 3D plant phenotyping in greenhouses, with scale restoration and semantic segmentation, achieving competitive accuracy to 3D scanners and improved robustness.

ABSTRACT

Accurate collection of plant phenotyping is critical to optimising sustainable farming practices in precision agriculture. Traditional phenotyping in controlled laboratory environments, while valuable, falls short in understanding plant growth under real-world conditions. Emerging sensor and digital technologies offer a promising approach for direct phenotyping of plants in farm environments. This study investigates a learning-based phenotyping method using the Neural Radiance Field to achieve accurate in-situ phenotyping of pepper plants in greenhouse environments. To quantitatively evaluate the performance of this method, traditional point cloud registration on 3D scanning data is implemented for comparison. Experimental result shows that NeRF(Neural Radiance Fields) achieves competitive accuracy compared to the 3D scanning methods. The mean distance error between the scanner-based method and the NeRF-based method is 0.865mm. This study shows that the learning-based NeRF method achieves similar accuracy to 3D scanning-based methods but with improved scalability and robustness.

Motivation & Objective

• Motivate accurate, in-situ plant phenotyping in real greenhouse environments.
• Evaluate a NeRF-based 3D reconstruction approach against high-precision 3D scanners for pepper plants.
• Address scale ambiguity in NeRF reconstructions through a scale restoration method.
• Integrate 3D semantic segmentation to improve phenotype detection and measurement.
• Assess robustness and generalization of NeRF-based phenotyping in agricultural scenes.

Proposed method

• Reconstruct 3D scenes from multi-view images using Neural Radiance Fields (NeRF).
• Adopt Instant-NGP and Neus variants to accelerate training and improve surface accuracy.
• Introduce a scale restoration pipeline using calibration plates to recover true model scale.
• Incorporate a 3D semantic segmentation network (PointNet-based) for phenotype extraction.
• Compare NeRF-based reconstructions with high-precision scanner point clouds via mean distance metrics.
• Evaluate reconstruction quality with PSNR and SSIM, and report phenotype measurements after scale calibration.

Experimental results

Research questions

• RQ1 How does NeRF-based 3D reconstruction compare to high-precision 3D scanners for pepper plant phenotyping in a greenhouse?
• RQ2 Can scale restoration recover true scene dimensions in NeRF reconstructions for accurate phenotyping?
• RQ3 What is the impact of incorporating a 3D semantic segmentation network on phenotype detection and measurement accuracy?
• RQ4 How do Fast NeRF variants (Instant-NGP, Instant-NSR) perform relative to traditional multi-view stereo (MVS) in this domain?

Key findings

• NeRF-based reconstruction achieves mean point-distance of 0.865 mm compared to scanner-based methods.
• Instant-NGP and Instant-NSR outperform previous multi-view reconstruction methods in PSNR/SSIM with faster training times; Instant-NSR(Ours) attains 28.74 dB PSNR and 0.810 SSIM.
• Scale restoration enables phenotypic measurements with sub-1% error after calibration; Instant-NSR achieves 0.094% difference vs 0.204% for Instant-NGP and 0.151% for the 3D scanner in the reported setup.
• Phenotyping measurements (height/width) after scale restoration show Instant-NSR achieving 0.094% deviation from true values, outperforming Instant-NGP (0.204%) and the scanner (0.151%).
• A 3D semantic segmentation network (PointNet-based) facilitates robust extraction of fruit regions for accurate phenotyping in cluttered greenhouse scenes.