[Paper Review] In-Field 3D Wheat Head Instance Segmentation From TLS Point Clouds Using Deep Learning Without Manual Labels

Tomislav Medić, Liangliang Nan|arXiv (Cornell University)|Mar 15, 2026

Smart Agriculture and AI0 citations

TL;DR

A two-stage pipeline uses zero-shot 2D foundations to propose 3D wheat-head instances from TLS point clouds and then trains a 3D panoptic-style network with pseudo-labels, achieving competitive 3D wheat head segmentation without manual annotations.

ABSTRACT

3D instance segmentation for laser scanning (LiDAR) point clouds remains a challenge in many remote sensing-related domains. Successful solutions typically rely on supervised deep learning and manual annotations, and consequently focus on objects that can be well delineated through visual inspection and manual labeling of point clouds. However, for tasks with more complex and cluttered scenes, such as in-field plant phenotyping in agriculture, such approaches are often infeasible. In this study, we tackle the task of in-field wheat head instance segmentation directly from terrestrial laser scanning (TLS) point clouds. To address the problem and circumvent the need for manual annotations, we propose a novel two-stage pipeline. To obtain the initial 3D instance proposals, the first stage uses 3D-to-2D multi-view projections, the Grounded SAM pipeline for zero-shot 2D object-centric segmentation, and multi-view label fusion. The second stage uses these initial proposals as noisy pseudo-labels to train a supervised 3D panoptic-style segmentation neural network. Our results demonstrate the feasibility of the proposed approach and show performance improvementsrelative to Wheat3DGS, a recent alternative solution for in-field wheat head instance segmentation without manual 3D annotations based on multi-view RGB images and 3D Gaussian Splatting, showcasing TLS as a competitive sensing alternative. Moreover, the results show that both stages of the proposed pipeline can deliver usable 3D instance segmentation without manual annotations, indicating promising, low-effort transferability to other comparable TLS-based point cloud segmentation tasks.

Motivation & Objective

• Address the challenge of 3D instance segmentation for complex in-field wheat heads in TLS point clouds without manual 3D annotations.
• Leverage 2D foundation models via multi-view projections to generate initial 3D proposals (Stage-I) and refine with a 3D panoptic-style network trained on pseudo-labels (Stage-II).
• Compare TLS-based segmentation against Wheat3DGS and assess zero-shot transferability to TLS data.
• Demonstrate that two-stage learning can yield usable 3D instance segmentation in cluttered canopy scenes.

Proposed method

• Stage-I: transform per-scan TLS point clouds into 2D range and intensity images via multi-view projection; apply Grounded-SAM with a text prompt to generate 2D instance masks; project masks back to 3D; fuse proposals with a graph-based clustering (kNN + IoU pruning, triangle-based weight) and Highly Connected Sub-graphs to produce 3D instance proposals.
• Stage-II: train a 3D panoptic-style network on pseudo-labels derived from Stage-I outputs using a Minkowski-Engine backbone with semantic, instance clustering, and discriminative-embedding heads; transfer Stage-I labels to a merged query point cloud and perform label refinement before training; merge sphere-level predictions into a single scene result.
• Interface between stages: create a merged point cloud Pt and a query cloud Pq; transfer Stage-I labels to Pt via nearest-neighbor interpolation within distance dp; refine labels with weighted voting and instance-wise voting to form Ptq for training and evaluation.
• Evaluation uses sparse ground-truth annotations (single 3D point per wheat head) due to occlusion; metrics include TP/FP/FN, precision, recall, F1, counting error (CE), and relative counting error (RCE).
• The pipeline follows prior methods for panoptic segmentation in large-scale LiDAR data, adapting to TLS data and using zero-shot 2D foundation models for initial proposals.

Figure 1: Flowchart of the implemented multi-view projection-based 3D instance segmentation (white: I/O, blue: data processing modules).

Experimental results

Research questions

• RQ1 Can zero-shot 2D foundation models (Grounded SAM) generate usable 3D wheat head proposals from TLS point clouds without manual 3D annotations?
• RQ2 Does a Stage-II 3D panoptic-style network trained on Stage-I pseudo-labels improve 3D wheat head instance segmentation in cluttered in-field TLS data?
• RQ3 How does the TLS-based pipeline compare to the Wheat3DGS image-based baseline for in-field wheat head instance segmentation?
• RQ4 Are Stage-I and Stage-II complementary, and does merging their outputs improve detection and counting performance?

Key findings

• TLS-based pipeline outperforms the Wheat3DGS image-based baseline on selected metrics, with higher F1-score in the hold-out test plot.
• Stage-II provides measurable improvements over Stage-I in all reported metrics, and combining Stage-I and Stage-II (I+II) yields the best recall and F1 among the variants.
• Stage-I and Stage-II detect different subsets of heads, with some instances detected by one stage and missed by the other, suggesting complementary strengths.
• Under-segmentation is a notable failure mode in dense clusters, and Stage-II struggles with out-of-distribution heads, indicating opportunities for improved augmentation and post-processing.
• The authors demonstrate a viable zero-shot 3D instance segmentation approach for TLS data, highlighting the potential transferability to other TLS-based segmentation tasks.

Figure 2: Examples of intensity (top) and range (bottom) images generated by the per-scan 3D-to-2D projection module, with a few examples of 2D instance masks generated by the 2D instance segmentation module. For clarity, only a small subset of the masks is shown.