[Paper Review] Fine-Grained Visual Classification of Plant Species In The Wild: Object Detection as A Reinforced Means of Attention

Matthew Keaton, Ram J. Zaveri+4 more|arXiv (Cornell University)|Jun 3, 2021

Species Distribution and Climate Change36 references6 citations

TL;DR

This paper proposes a bottom-up approach for fine-grained plant species identification in the wild by using object detection to localize key plant organs (leaves, flowers, fruits, stems, HDL) and fusing predictions from organ-specific classifiers. The method achieves 79.36% accuracy using a sum rule fusion, outperforming both individual organ classifiers and a full-image ResNet-18 baseline, while introducing a publicly available long-tail dataset for organ detection and organ-based classification.

ABSTRACT

Plant species identification in the wild is a difficult problem in part due to the high variability of the input data, but also because of complications induced by the long-tail effects of the datasets distribution. Inspired by the most recent fine-grained visual classification approaches which are based on attention to mitigate the effects of data variability, we explore the idea of using object detection as a form of attention. We introduce a bottom-up approach based on detecting plant organs and fusing the predictions of a variable number of organ-based species classifiers. We also curate a new dataset with a long-tail distribution for evaluating plant organ detection and organ-based species identification, which is publicly available.

Motivation & Objective

• To address the challenge of high data variability and long-tail class distributions in fine-grained plant species identification from wild images.
• To improve robustness against background noise and intra-class variance by localizing discriminative plant organs using object detection.
• To develop a fusion framework that combines predictions from multiple organ-based classifiers to enhance overall species identification accuracy.
• To create and release a new long-tail dataset for evaluating plant organ detection and organ-based species classification.

Proposed method

• A bottom-up approach uses an object detector to localize plant organs (leaves, flowers, fruits, stems, high-density leaves) in wild images.
• Each detected region of interest (ROI) is passed to a corresponding organ-based species classifier trained on organ-specific features.
• A fusion strategy—sum rule, product rule, or voting—is applied to aggregate per-organ classification probabilities into a final image-level prediction.
• The method leverages supervised object detection to reduce reliance on unsupervised attention, improving robustness in long-tailed settings.
• A new dataset with 1,000 species and long-tail distributions was curated, including 3568 leaf samples and 1122 fruit samples, with annotations for five organ types.
• Evaluation uses COCO-style average precision (AP) for detection and accuracy for classification, with ablation on fusion rules.

Experimental results

Research questions

• RQ1 Can object detection serve as a more robust, supervised form of attention for fine-grained plant species classification in the wild?
• RQ2 How does fusing predictions from multiple organ-based classifiers improve overall species identification accuracy compared to single-image or single-organ approaches?
• RQ3 To what extent does the long-tail distribution of real-world plant image data affect the performance of organ detection and organ-based classification?
• RQ4 Can a bottom-up, organ-localization approach mitigate the negative effects of data variability and background clutter in wild plant image recognition?

Key findings

• The proposed method achieved 79.36% species identification accuracy using the sum rule fusion, outperforming the ResNet-18 baseline of 69.65%.
• The sum rule fusion strategy yielded the highest accuracy among the three fusion methods tested, demonstrating effective aggregation of multiple organ predictions.
• Organ-based classifiers achieved high accuracy on abundant organs (e.g., 75.24% for flowers), but performance dropped significantly for rare organs like stems (58.24%) and HDL (34.21%), reflecting data scarcity.
• The organ detector achieved an overall AP of 42.9% on the new dataset, with higher performance on leaves (40.9%) and stems (74.0%), indicating challenges in detecting smaller or less distinct organs.
• The dataset exhibits a pronounced long-tail distribution, with 161 species having complete ROI data and at least 130 leaf samples, but many species having very few organ annotations.
• Despite moderate detection performance, the fusion mechanism effectively filtered false positives and leveraged unannotated detections, maintaining high final accuracy.