[Paper Review] Eff-3DPSeg: 3D organ-level plant shoot segmentation using annotation-efficient point clouds

Liyi Luo, Xintong Jiang+5 more|arXiv (Cornell University)|Dec 20, 2022

Smart Agriculture and AI4 citations

TL;DR

This paper proposes Eff-3DPSeg, a weakly supervised 3D plant shoot segmentation framework that leverages self-supervised pretraining with viewpoint bottleneck loss and fine-tuning using only 0.5% annotated points. It achieves high performance—95.1% precision, 96.6% recall, and 92.2% mIoU—on stem and leaf segmentation, enabling efficient organ-level phenotyping with minimal annotation effort.

ABSTRACT

Reliable and automated 3D plant shoot segmentation is a core prerequisite for the extraction of plant phenotypic traits at the organ level. Combining deep learning and point clouds can provide effective ways to address the challenge. However, fully supervised deep learning methods require datasets to be point-wise annotated, which is extremely expensive and time-consuming. In our work, we proposed a novel weakly supervised framework, Eff-3DPSeg, for 3D plant shoot segmentation. First, high-resolution point clouds of soybean were reconstructed using a low-cost photogrammetry system, and the Meshlab-based Plant Annotator was developed for plant point cloud annotation. Second, a weakly-supervised deep learning method was proposed for plant organ segmentation. The method contained: (1) Pretraining a self-supervised network using Viewpoint Bottleneck loss to learn meaningful intrinsic structure representation from the raw point clouds; (2) Fine-tuning the pre-trained model with about only 0.5% points being annotated to implement plant organ segmentation. After, three phenotypic traits (stem diameter, leaf width, and leaf length) were extracted. To test the generality of the proposed method, the public dataset Pheno4D was included in this study. Experimental results showed that the weakly-supervised network obtained similar segmentation performance compared with the fully-supervised setting. Our method achieved 95.1%, 96.6%, 95.8% and 92.2% in the Precision, Recall, F1-score, and mIoU for stem leaf segmentation and 53%, 62.8% and 70.3% in the AP, AP@25, and AP@50 for leaf instance segmentation. This study provides an effective way for characterizing 3D plant architecture, which will become useful for plant breeders to enhance selection processes.

Motivation & Objective

• To reduce the high annotation cost of fully supervised 3D plant shoot segmentation using point clouds.
• To develop a weakly supervised deep learning framework that achieves high segmentation accuracy with minimal human-annotated data.
• To enable reliable extraction of organ-level phenotypic traits such as stem diameter, leaf width, and leaf length.
• To validate the method’s generalization on public datasets like Pheno4D.
• To provide a scalable solution for plant breeders to accelerate phenotyping workflows.

Proposed method

• Reconstructed high-resolution 3D point clouds of soybean shoots using a low-cost photogrammetry system.
• Developed a Meshlab-based Plant Annotator tool for efficient point cloud annotation.
• Pretrained a deep neural network using a self-supervised loss—Viewpoint Bottleneck loss—to learn intrinsic structural representations from raw point clouds.
• Fine-tuned the pre-trained model using only 0.5% of point-wise annotations to perform organ-level segmentation.
• Applied the trained model to extract three phenotypic traits: stem diameter, leaf width, and leaf length.
• Evaluated the method on both custom soybean data and the public Pheno4D dataset to assess generalization.

Experimental results

Research questions

• RQ1 Can self-supervised pretraining on raw 3D point clouds reduce the need for extensive human annotation in plant shoot segmentation?
• RQ2 How does the performance of a weakly supervised segmentation model compare to fully supervised baselines when using only 0.5% annotated points?
• RQ3 To what extent does the proposed method generalize across different plant species and datasets, such as Pheno4D?
• RQ4 Can the segmentation model reliably extract key phenotypic traits like stem diameter and leaf dimensions?
• RQ5 Does the Viewpoint Bottleneck loss effectively capture structural semantics in plant point clouds without supervision?

Key findings

• The weakly supervised model achieved 95.1% precision, 96.6% recall, and 92.2% mIoU on stem and leaf segmentation, closely matching fully supervised performance.
• For leaf instance segmentation, the model achieved 70.3% AP@50, 62.8% AP@25, and 53% AP, indicating strong instance-level detection capability.
• The method generalized well to the public Pheno4D dataset, demonstrating robustness across different plant architectures and imaging conditions.
• The use of only 0.5% annotated points significantly reduced annotation cost while maintaining high segmentation accuracy.
• Phenotypic trait extraction (stem diameter, leaf width, leaf length) was successfully enabled by the segmentation pipeline.
• Self-supervised pretraining with Viewpoint Bottleneck loss effectively learned meaningful structural representations from raw point clouds, improving downstream fine-tuning efficiency.