[Paper Review] Deep learning powered real-time identification of insects using citizen science data

Shivani Chiranjeevi, Mojdeh Sadaati+11 more|arXiv (Cornell University)|Jun 4, 2023

Insect and Arachnid Ecology and Behavior11 citations

TL;DR

InsectNet uses a large citizen-science insect image dataset with self-supervised learning, transfer learning, OOD detection, and conformal prediction to identify 2526 insect species with over 96% accuracy and provides a public web portal and reusable ML stack.

ABSTRACT

Insect-pests significantly impact global agricultural productivity and quality. Effective management involves identifying the full insect community, including beneficial insects and harmful pests, to develop and implement integrated pest management strategies. Automated identification of insects under real-world conditions presents several challenges, including differentiating similar-looking species, intra-species dissimilarity and inter-species similarity, several life cycle stages, camouflage, diverse imaging conditions, and variability in insect orientation. A deep-learning model, InsectNet, is proposed to address these challenges. InsectNet is endowed with five key features: (a) utilization of a large dataset of insect images collected through citizen science; (b) label-free self-supervised learning for large models; (c) improving prediction accuracy for species with a small sample size; (d) enhancing model trustworthiness; and (e) democratizing access through streamlined MLOps. This approach allows accurate identification (>96% accuracy) of over 2500 insect species, including pollinator (e.g., butterflies, bees), parasitoid (e.g., some wasps and flies), predator species (e.g., lady beetles, mantises, dragonflies) and harmful pest species (e.g., armyworms, cutworms, grasshoppers, stink bugs). InsectNet can identify invasive species, provide fine-grained insect species identification, and work effectively in challenging backgrounds. It also can abstain from making predictions when uncertain, facilitating seamless human intervention and making it a practical and trustworthy tool. InsectNet can guide citizen science data collection, especially for invasive species where early detection is crucial. Similar approaches may transform other agricultural challenges like disease detection and underscore the importance of data collection, particularly through citizen science efforts..

Motivation & Objective

• Harness citizen science data to build a scalable, accurate insect classifier.
• Handle metamorphosis and fine-grained species with high intra- and inter-species variability.
• Improve reliability and trust through OOD detection and conformal prediction.
• Democratize access via web portal and reusable MLOps workflows.

Proposed method

• Assemble a dataset of 6M images from 2526 insect species from iNaturalist (subset of 13M insects, 100k species).
• Apply label-free self-supervised learning to pre-train models on unlabeled data, followed by finetuning on labeled data.
• Use AlphaNet to boost accuracy for low-sample species without retraining the base model.
• Wrap the classifier with energy-based OOD detection to flag uncertain or out-of-distribution images.
• Employ conformal predictions to output prediction sets with guaranteed confidence (≥97.5%).
• Provide public access to the model, weights, quantized versions, and MLOps workflows via a web portal.

Figure 1: InsectNet in action. After an image is uploaded, InsectNet first performs out-of-distribution (OOD) detection. (Left) If OOD detection is true, InsectNet provides a warning along with its prediction. (Middle) If not OOD, InsectNet produces a prediction with no warning. (Right) Additionally

Experimental results

Research questions

• RQ1 Can InsectNet accurately identify 2526 insect species across life stages using citizen science images?
• RQ2 Does self-supervised pre-training on large unlabeled datasets improve final species classification performance?
• RQ3 Can a post hoc wrapper (AlphaNet) improve low-sample species accuracy without harming overall accuracy?
• RQ4 Does energy-based OOD detection and conformal prediction enhance trust and safe human intervention?
• RQ5 Is the approach deployable via democratized access (web portal and MLOps) for practical use in IPM and CSPM?

Key findings

• InsectNet achieves 96.4% classification accuracy across 2526 species; mean per-species accuracy is 94%.
• Only 3.40% of species have accuracy below 80%, with most underrepresented species (>1000 images) performing well.
• SSL pre-training with large unlabeled data and multi-dataset pre-training campaigns significantly boost representation quality.
• AlphaNet improves low-sample species accuracy from 79.7% to 87.6% with only a 1.3% drop in overall accuracy (96.4% to 95.1%).
• Out-of-distribution detection and conformal prediction provide fail-safes and confidence guarantees (≥97.5%), enabling trusted human intervention.
• InsectNet can identify insects across metamorphosis stages and in challenging backgrounds, including invasive and beneficial species.

Figure 2: InsectNet is able to accurately identify insect species across the life cycle stages. Top left: charismatic species Danaus plexippus (Monarch butterfly), Bottom left: an invasive species Lycorma delicatula (Spotted lanternfly). Right panel: Examples of the ability of InsectNet to accuratel