[Paper Review] Accurate Fruit Localisation for Robotic Harvesting using High Resolution LiDAR-Camera Fusion

Hanwen Kang, Xing Wang+1 more|arXiv (Cornell University)|May 1, 2022

Smart Agriculture and AI4 citations

TL;DR

This paper proposes a LiDAR-camera fusion method using high-resolution solid-state LiDAR and RGB cameras for accurate fruit localization in apple orchards. By applying state-of-the-art target-based and targetless extrinsic calibration, the system fuses point clouds and images to enable one-stage instance segmentation, achieving sub-0.3 cm standard deviation in fruit localization—five times more accurate than the Realsense D455 at all tested distances.

ABSTRACT

Accurate depth-sensing plays a crucial role in securing a high success rate of robotic harvesting in natural orchard environments. Solid-state LiDAR (SSL), a recently introduced LiDAR technique, can perceive high-resolution geometric information of the scenes, which can be potential utilised to receive accurate depth information. Meanwhile, the fusion of the sensory information from LiDAR and camera can significantly enhance the sensing ability of the harvesting robots. This work introduces a LiDAR-camera fusion-based visual sensing and perception strategy to perform accurate fruit localisation for a harvesting robot in the apple orchards. Two SOTA extrinsic calibration methods, target-based and targetless-based, are applied and evaluated to obtain the accurate extrinsic matrix between the LiDAR and camera. With the extrinsic calibration, the point clouds and color images are fused to perform fruit localisation using a one-stage instance segmentation network. Experimental shows that LiDAR-camera achieves better quality on visual sensing in the natural environments. Meanwhile, introducing the LiDAR-camera fusion largely improves the accuracy and robustness of the fruit localisation. Specifically, the standard deviations of fruit localisation by using LiDAR-camera at 0.5 m, 1.2 m, and 1.8 m are 0.245, 0.227, and 0.275 cm respectively. These measurement error is only one one fifth of that from Realsense D455. Lastly, we have attached our visualised point cloud to demonstrate the highly accurate sensing method.

Motivation & Objective

• To improve fruit localization accuracy in unstructured orchard environments for robotic harvesting.
• To address the limitations of stereo depth cameras like the Realsense D455 in complex, natural outdoor scenes.
• To achieve precise extrinsic calibration between LiDAR and camera sensors using both target-based and targetless methods.
• To fuse high-resolution LiDAR point clouds with RGB images for enhanced 3D perception and semantic understanding.
• To demonstrate superior depth sensing and localization robustness in real-world orchard conditions.

Proposed method

• Utilizes solid-state LiDAR (SSL) to capture high-resolution 3D geometric data of orchard scenes.
• Employs two state-of-the-art extrinsic calibration methods—target-based and targetless-based—using feature matching and reprojection error minimization.
• Fuses calibrated LiDAR point clouds with RGB images to generate colorized 3D point clouds with semantic context.
• Applies a one-stage instance segmentation network to localize fruits in the fused 3D-2D space.
• Validates calibration accuracy and localization performance across multiple distances: 0.5 m, 1.2 m, and 1.8 m.
• Uses visualized point clouds to demonstrate the quality and accuracy of the sensing system.

Experimental results

Research questions

• RQ1 Can high-resolution solid-state LiDAR improve depth sensing accuracy in natural orchard environments compared to conventional stereo cameras?
• RQ2 How does LiDAR-camera fusion enhance fruit localization accuracy and robustness in unstructured outdoor settings?
• RQ3 What is the comparative performance of target-based versus targetless extrinsic calibration for LiDAR-camera systems in orchard applications?
• RQ4 To what extent does the fusion of geometric and semantic data reduce localization error in robotic harvesting tasks?
• RQ5 How does the standard deviation of fruit localization vary with distance when using LiDAR-camera fusion versus stereo depth sensors?

Key findings

• The LiDAR-camera fusion system achieved standard deviations of 0.245 cm, 0.227 cm, and 0.275 cm in fruit localization at 0.5 m, 1.2 m, and 1.8 m distances, respectively.
• These localization errors were only about one-fifth of those from the Realsense D455, which showed increasing standard deviation with distance.
• The targetless-based calibration method offered comparable accuracy to the target-based method but with greater convenience and even feature distribution in the scene.
• LiDAR-camera fusion produced higher-quality, less distorted point clouds than the Realsense D455, especially in capturing fine geometries like branches and fruits.
• The center region of the LiDAR point cloud exhibited higher accuracy than the edges due to laser beam divergence, suggesting multi-view LiDAR fusion could further improve performance.
• The system demonstrated superior robustness and accuracy in complex, unstructured orchard environments under varying illumination and scene complexity.