[Paper Review] Out-of-distribution detection algorithms for robust insect classification

Mojdeh Saadati, Aditya Balu+6 more|arXiv (Cornell University)|May 2, 2023

Smart Agriculture and AIAgricultural and Biological Sciences3 citations

TL;DR

This paper evaluates out-of-distribution (OOD) detection algorithms—Maximum Softmax Probability (MSP), Mahalanobis distance (MAH), and Energy-Based Models (EBM)—to improve robustness in deep learning-based insect classification for agricultural applications. EBM outperformed others, especially under data imbalance, enabling models to abstain from uncertain predictions and enhancing trust in real-world deployment.

ABSTRACT

Deep learning-based approaches have produced models with good insect classification accuracy; Most of these models are conducive for application in controlled environmental conditions. One of the primary emphasis of researchers is to implement identification and classification models in the real agriculture fields, which is challenging because input images that are wildly out of the distribution (e.g., images like vehicles, animals, humans, or a blurred image of an insect or insect class that is not yet trained on) can produce an incorrect insect classification. Out-of-distribution (OOD) detection algorithms provide an exciting avenue to overcome these challenge as it ensures that a model abstains from making incorrect classification prediction of non-insect and/or untrained insect class images. We generate and evaluate the performance of state-of-the-art OOD algorithms on insect detection classifiers. These algorithms represent a diversity of methods for addressing an OOD problem. Specifically, we focus on extrusive algorithms, i.e., algorithms that wrap around a well-trained classifier without the need for additional co-training. We compared three OOD detection algorithms: (i) Maximum Softmax Probability, which uses the softmax value as a confidence score, (ii) Mahalanobis distance-based algorithm, which uses a generative classification approach; and (iii) Energy-Based algorithm that maps the input data to a scalar value, called energy. We performed an extensive series of evaluations of these OOD algorithms across three performance axes: (a) extit{Base model accuracy}: How does the accuracy of the classifier impact OOD performance? (b) How does the extit{level of dissimilarity to the domain} impact OOD performance? and (c) extit{Data imbalance}: How sensitive is OOD performance to the imbalance in per-class sample size?

Motivation & Objective

• Address the challenge of unreliable insect classification when models encounter out-of-distribution (OOD) inputs such as non-insects or untrained species in real-world agricultural settings.
• Evaluate the robustness of state-of-the-art OOD detection algorithms—MSP, MAH, and EBM—on a large-scale insect classification model trained on over 2 million images.
• Assess how model accuracy, OOD input dissimilarity, and data imbalance affect OOD detection performance to guide practical deployment in agriculture.
• Provide actionable guidelines for deploying reliable, trustworthy insect classification systems in field conditions where data distribution shifts are common.

Proposed method

• Trained a deep learning classifier on 142 agriculturally relevant insect species using over 2 million images from diverse sources, including iNaturalist.
• Applied three OOD detection algorithms: (1) Maximum Softmax Probability (MSP), using the highest softmax probability as a confidence score; (2) Mahalanobis distance (MAH), based on generative modeling of class-conditional feature distributions; and (3) Energy-Based Model (EBM), which assigns a scalar energy score to inputs.
• Evaluated OOD performance across three axes: base model accuracy, level of input dissimilarity to training data, and data imbalance in training classes.
• Used AUROC (Area Under the ROC Curve) as the primary metric to quantify OOD detection performance across different OOD datasets, including non-insect images and unbalanced class distributions.
• Generated synthetic data with varying class imbalances to simulate real-world agricultural data distributions and assess algorithm sensitivity.
• Compared OOD detection performance across three in-distribution (ID) subsets: balanced, unbalanced, and uniformly sampled from the same distribution as the original dataset.

Figure 1 : Out-of-distribution visualization in insect classification. The OOD classifier tries to define decision boundaries for each insect while ensuring no non-insect images fall into any known insect classes. There is a spectrum of OOD images. Here, we distinguish them as contextually near OOD

Experimental results

Research questions

• RQ1 How does the base model accuracy correlate with OOD detection performance across different OOD detection algorithms?
• RQ2 How does the degree of dissimilarity between OOD inputs and the training distribution affect the performance of OOD detection algorithms?
• RQ3 How sensitive are OOD detection algorithms to data imbalance in the training data, particularly in the context of agricultural insect classification?
• RQ4 Which OOD detection algorithm demonstrates the most robust performance across varying levels of data imbalance and input distribution shifts?

Key findings

• The Energy-Based Model (EBM) achieved the highest OOD detection performance, with consistently high AUROC across all evaluation conditions, especially under data imbalance.
• MSP showed strong OOD detection performance and was robust to data imbalance, indicating its suitability for real-world agricultural applications with uneven class distributions.
• The Mahalanobis distance (MAH) algorithm was sensitive to data imbalance, with significantly degraded performance on unbalanced datasets, highlighting a key limitation in practical deployment.
• For models with fewer parameters, there was a positive correlation between base classifier accuracy and OOD detection performance; this correlation disappeared in more complex models.
• EBM demonstrated superior robustness when OOD inputs were more similar to the training or pretraining data distribution, suggesting better generalization under realistic distribution shifts.
• The results indicate that OOD detection enables models to abstain from making predictions on uncertain or out-of-distribution inputs, thereby improving reliability and trust in autonomous agricultural systems.

Figure 2 : The analysis process: Box 1 represents the data. We split the ID-distribution data into three sub-datasets that are used for training the insect classifier, training the Mahalanobis distance OOD algorithm, and testing the results. We also utilize four different OOD datasets for testing ou