[Paper Review] PLANesT-3D: A new annotated dataset for segmentation of 3D plant point clouds

Kerem Mertoğlu, Yusuf Şalk+7 more|arXiv (Cornell University)|Jul 30, 2024

Remote Sensing and LiDAR Applications4 citations

TL;DR

This paper introduces PLANesT-3D, a new public dataset of 34 high-resolution, color 3D point clouds from three plant species—pepper, rose, and currant—annotated with semantic (leaf/stem) and instance-level labels. The study evaluates deep learning models including PointNet++, RoseSegNet, and a novel method SP-LSCnet, with RoseSegNet achieving the highest mean Intersection over Union (MIoU) of 95.5% on pepper plants, demonstrating the dataset's value for advancing 3D plant segmentation in automated phenotyping.

ABSTRACT

Creation of new annotated public datasets is crucial in helping advances in 3D computer vision and machine learning meet their full potential for automatic interpretation of 3D plant models. Despite the proliferation of deep neural network architectures for segmentation and phenotyping of 3D plant models in the last decade, the amount of data, and diversity in terms of species and data acquisition modalities are far from sufficient for evaluation of such tools for their generalization ability. To contribute to closing this gap, we introduce PLANesT-3D; a new annotated dataset of 3D color point clouds of plants. PLANesT-3D is composed of 34 point cloud models representing 34 real plants from three different plant species: extit{Capsicum annuum}, extit{Rosa kordana}, and extit{Ribes rubrum}. Both semantic labels in terms of "leaf" and "stem", and organ instance labels were manually annotated for the full point clouds. PLANesT-3D introduces diversity to existing datasets by adding point clouds of two new species and providing 3D data acquired with the low-cost SfM/MVS technique as opposed to laser scanning or expensive setups. Point clouds reconstructed with SfM/MVS modality exhibit challenges such as missing data, variable density, and illumination variations. As an additional contribution, SP-LSCnet, a novel semantic segmentation method that is a combination of unsupervised superpoint extraction and a 3D point-based deep learning approach is introduced and evaluated on the new dataset. The advantages of SP-LSCnet over other deep learning methods are its modular structure and increased interpretability. Two existing deep neural network architectures, PointNet++ and RoseSegNet, were also tested on the point clouds of PLANesT-3D for semantic segmentation.

Motivation & Objective

• To address the scarcity of publicly available, high-quality 3D plant point cloud datasets with detailed semantic and instance annotations.
• To enable benchmarking and training of deep learning models for 3D plant segmentation across diverse plant species.
• To evaluate the performance of state-of-the-art deep learning architectures and a novel method, SP-LSCnet, on a new, challenging 3D plant segmentation benchmark.
• To support future research in transfer learning, small-sample learning, and the integration of color information in 3D plant analysis.

Proposed method

• The dataset was created using structure-from-motion (SfM) and multi-view stereo (MVS) reconstruction from hundreds of RGB images captured under controlled conditions.
• Manual annotation was performed to label all points as either 'leaf' or 'stem' at the semantic level, and individual organ instances were labeled at the instance level.
• A novel method, SP-LSCnet, was proposed, combining unsupervised superpoint extraction with a 3D point-based deep learning network for improved segmentation.
• Three deep learning models—PointNet++, RoseSegNet, and SP-LSCnet—were trained and evaluated on the PLANesT-3D dataset using mean Intersection over Union (MIoU) as the primary metric.
• Post-processing techniques such as label smoothing and graph-based re-evaluation were applied to improve segmentation results across all models.
• The dataset includes 34 full 3D point clouds with 600K to 5.5M points each, captured with high-resolution RGB cameras and reconstructed into dense, colorized point clouds.

Experimental results

Research questions

• RQ1 How does the performance of existing 3D semantic segmentation models compare on a new, diverse, and high-resolution 3D plant point cloud dataset?
• RQ2 To what extent can a novel method combining superpoint extraction and deep learning (SP-LSCnet) improve segmentation accuracy on complex plant structures?
• RQ3 How does the inclusion of color information in 3D point clouds affect segmentation performance, and what is its potential for future model development?
• RQ4 Can models trained on one plant species generalize to others, and what does this imply for transfer learning in plant phenotyping?

Key findings

• RoseSegNet achieved the highest mean Intersection over Union (MIoU) of 95.5% on pepper plants, 92.0% on rose plants, and 95.1% on currant plants, outperforming other models.
• SP-LSCnet achieved competitive results with MIoU scores of 95.0% on pepper, 89.8% on rose, and 94.5% on currant plants, demonstrating the effectiveness of its hybrid superpoint and deep learning approach.
• PointNet++ showed lower performance compared to RoseSegNet and SP-LSCnet, particularly on complex and dense plant structures, indicating limitations in handling variable point density and fine-grained geometry.
• All models struggled with small, isolated, or overlapping plant parts, especially in regions with low point density or high structural complexity.
• The dataset's high-resolution, colorized point clouds and detailed annotations provide a robust foundation for training and evaluating future 3D plant segmentation models.
• The availability of three different plant species enables future research into cross-species transfer learning and generalization of deep learning models in plant phenotyping.