[Paper Review] Robust Pollen Imagery Classification with Generative Modeling and Mixup Training

Jaideep Vitthal Murkute|arXiv (Cornell University)|Feb 25, 2021

Allergic Rhinitis and Sensitization9 references4 citations

TL;DR

This paper proposes a robust deep learning framework for pollen grain image classification by combining variational autoencoder (VAE)-based generative modeling, manifold mixup training, and EfficientNet backbones. The approach improves generalization through joint supervised and unsupervised learning, data augmentation, and model ensembling, achieving a 0.972578 weighted F1-score in the ICPR-2020 challenge—ranking fourth overall.

ABSTRACT

Deep learning approaches have shown great success in image classification tasks and can aid greatly towards the fast and reliable classification of pollen grain aerial imagery. However, often-times deep learning methods in the setting of natural images can suffer generalization problems and yield poor performance on unseen test distribution. In this work, we present and a robust deep learning framework that can generalize well for pollen grain aerobiological imagery classification. We develop a convolutional neural network-based pollen grain classification approach and combine some of the best practices in deep learning for better generalization. In addition to commonplace approaches like data-augmentation and weight regularization, we utilize implicit regularization methods like manifold mixup to allow learning of smoother decision boundaries. We also make use of proven state-of-the-art architectural choices like EfficientNet convolutional neural networks. Inspired by the success of generative modeling with variational autoencoders, we train models with a richer learning objective which can allow the model to focus on the relevant parts of the image. Finally, we create an ensemble of neural networks, for the robustness of the test set predictions. Based on our experiments, we show improved generalization performance as measured with a weighted F1-score with the aforementioned approaches. The proposed approach earned a fourth-place in the final rankings in the ICPR-2020 Pollen Grain Classification Challenge; with a 0.972578 weighted F1 score,0.950828 macro average F1 scores, and 0.972877 recognition accuracy.

Motivation & Objective

• To address generalization failures in deep learning models for pollen grain image classification under distribution shift and class imbalance.
• To improve model robustness by integrating unsupervised disentangled feature learning via variational autoencoders (VAEs).
• To enhance decision boundary smoothness and generalization using manifold mixup training.
• To achieve state-of-the-art performance on the ICPR-2020 Pollen Grain Classification Challenge through architectural and training innovations.

Proposed method

• Trained a VAE-based framework where the encoder extracts features and generates a latent code, and the decoder reconstructs the pollen grain patch to focus on relevant image regions.
• Optimized the VAE using the evidence lower bound (ELBO) objective, combining reconstruction loss (MSE) and KL divergence between posterior and prior distributions.
• Applied manifold mixup by interpolating features at random layers of the network, with hyperparameter α = 0.5 or 1.0, to encourage smoother decision boundaries.
• Employed a data augmentation pipeline including random rotation, zoom, flip, and crop transformations, with ImageNet normalization.
• Used an ensemble of four models—two EfficientNet-B3 and two EfficientNet-B4—trained with different mixup settings and architectures, with final predictions averaged.
• Fine-tuned the model in two stages: first freezing the encoder and training the classifier and decoder, then unfreezing all layers for joint optimization.

Experimental results

Research questions

• RQ1 Can combining VAE-based generative modeling with mixup improve generalization in pollen grain image classification?
• RQ2 How does the integration of unsupervised VAE learning affect performance on a downstream classification task with limited labeled data?
• RQ3 Does manifold mixup with varying α values lead to more robust and smoother decision boundaries in pollen classification?
• RQ4 To what extent do architectural choices like EfficientNet-B3/B4 and data augmentation contribute to improved generalization?
• RQ5 Can model ensembling further stabilize predictions and improve generalization beyond individual model performance?

Key findings

• The proposed method achieved a weighted F1-score of 0.972578 on the test set, ranking fourth in the ICPR-2020 Pollen Grain Classification Challenge.
• The ensemble of four VAE-based models achieved a test accuracy of 0.972877 and a macro F1-score of 0.950828.
• Models trained with mixup (α = 0.5 or 1.0) and VAE reconstruction showed consistent improvements over baseline EfficientNet models, with the best individual model (EffNet-B4 VAE, α=0.5) achieving 0.96909 weighted F1.
• The VAE component improved feature disentanglement and model focus on relevant image regions, even without explicit segmentation.
• The two-stage training strategy—freezing the encoder initially—led to better convergence and improved generalization.
• The ablation study showed that explicit segmentation of pollen grains was not beneficial, as modern CNNs with augmentation and dropout already handle background variations effectively.