[Paper Review] Deep Leaf Segmentation Using Synthetic Data

Daniel B. Ward, Peyman Moghadam+1 more|arXiv (Cornell University)|Jul 28, 2018

Smart Agriculture and AI11 references73 citations

TL;DR

The paperAugments limited real training data for leaf instance segmentation with synthetic Arabidopsis leaves generated via a Blender-based pipeline, achieving state-of-the-art results on CVPPP LSC, especially when combining real and synthetic data.

ABSTRACT

Automated segmentation of individual leaves of a plant in an image is a prerequisite to measure more complex phenotypic traits in high-throughput phenotyping. Applying state-of-the-art machine learning approaches to tackle leaf instance segmentation requires a large amount of manually annotated training data. Currently, the benchmark datasets for leaf segmentation contain only a few hundred labeled training images. In this paper, we propose a framework for leaf instance segmentation by augmenting real plant datasets with generated synthetic images of plants inspired by domain randomisation. We train a state-of-the-art deep learning segmentation architecture (Mask-RCNN) with a combination of real and synthetic images of Arabidopsis plants. Our proposed approach achieves 90% leaf segmentation score on the A1 test set outperforming the-state-of-the-art approaches for the CVPPP Leaf Segmentation Challenge (LSC). Our approach also achieves 81% mean performance over all five test datasets.

Motivation & Objective

• Address the scarcity of labeled data for leaf instance segmentation in plant phenotyping.
• Propose a synthetic data generation pipeline for Arabidopsis leaves leveraging domain randomisation.
• Evaluate the impact of real, synthetic, and mixed training data on CVPPP Leaf Segmentation Challenge results.

Proposed method

• Use Mask-RCNN with a ResNet101 backbone trained on a mix of real CVPPP A1 data and 10,000 synthetic plant images.
• Generate synthetic Arabidopsis plants by designing a single inspiration leaf in Blender and deforming, texturing, and placing leaves to form rosettes.
• Render synthetic leaves with varied textures (30 textures) and backgrounds (random plant pot backgrounds) and apply Gaussian blur to blend with backgrounds.
• Train with balanced batches of real and synthetic data and apply standard data augmentation (flips, rotations, zoom, crop).
• Experiment with freezing the feature extractor to compare with domain randomisation approaches.

Experimental results

Research questions

• RQ1 Can synthetic plant images improve leaf instance segmentation when combined with real data for CVPPP LSC?
• RQ2 How does texture variation and background randomisation in synthetic data affect segmentation performance?
• RQ3 Does freezing pre-trained features interact positively or negatively with synthetic data for deformable plant models?
• RQ4 What are the limits of synthetic data in matching real leaf geometry across species (Arabidopsis vs tobacco)?

Key findings

• Training with both real and synthetic data (Synthetic+real) achieves 90.0 SBD on A1, 81.0 on A2, 51.0 on A3, 88 on A4, and 82 on A5, averaging 81 across all five test sets.
• Synthetic data alone generally underperforms compared to real data on A1, A2, A4, but exceeds real data on some sets when texture variation is controlled.
• Per-leaf texture (Synthetic-leaf) improves performance across most test sets except A3, illustrating the importance of texture realism for deformable leaves.
• Randomised textures sampled from COCO (Synthetic-COCO) performs worst, highlighting the need for domain-appropriate textures.
• Mixing real and synthetic data provides the best overall generalisation, improving robustness and reducing false positives from background artifacts (e.g., fiducial markers).
• Freezing the feature extractor when training on synthetic data degrades results in this study, suggesting deformable plant geometry challenges the rigid-object assumption of some domain adaptation approaches.