[Paper Review] An evaluation of pre-trained models for feature extraction in image classification

Erick da Silva Puls, Matheus Vinícius Todescato+1 more|arXiv (Cornell University)|Oct 3, 2023

COVID-19 diagnosis using AI8 citations

TL;DR

The paper systematically evaluates 16 pre-trained models as feature extractors across four image datasets, finding transformer-based models (especially CLIP-ViT-B, CLIP-ResNet50, and VisionTransformer-H/14) generally outperform CNN-only architectures.

ABSTRACT

In recent years, we have witnessed a considerable increase in performance in image classification tasks. This performance improvement is mainly due to the adoption of deep learning techniques. Generally, deep learning techniques demand a large set of annotated data, making it a challenge when applying it to small datasets. In this scenario, transfer learning strategies have become a promising alternative to overcome these issues. This work aims to compare the performance of different pre-trained neural networks for feature extraction in image classification tasks. We evaluated 16 different pre-trained models in four image datasets. Our results demonstrate that the best general performance along the datasets was achieved by CLIP-ViT-B and ViT-H-14, where the CLIP-ResNet50 model had similar performance but with less variability. Therefore, our study provides evidence supporting the choice of models for feature extraction in image classification tasks.

Motivation & Objective

• Motivate transfer learning for image classification when labeled data is limited.
• Compare multiple pre-trained models as fixed feature extractors across diverse datasets.
• Identify which architectures (CNN vs. transformer-based) give best general performance in FE tasks.
• Provide guidance on model choice for feature extraction in image classification tasks.

Proposed method

• Freeze pre-trained backbone networks and replace the final classifier layer with a new output layer sized to dataset classes.
• Apply consistent pre-processing (resize, center crop, RGB conversion) and train with a 5-fold cross-validation setup.
• Train with learning rate 0.001, momentum 0.9, Adam optimizer, Cross-Entropy loss, for up to 100 epochs with early stopping.
• Evaluate using accuracy, macro F1, and weighted F1 across folds.

Figure 1: Illustration of the imbalance in the number of images for each class on the Geological Images dataset [ 9 ] .

Experimental results

Research questions

• RQ1 Which pre-trained models yield the best accuracy and F1 metrics when used as fixed feature extractors across the chosen datasets?
• RQ2 How do CNN-based architectures compare to transformer-based architectures in feature extraction performance and variability?
• RQ3 Do certain datasets (e.g., those with more classes or heterogeneous image sizes) influence the relative effectiveness of different pre-trained models?
• RQ4 What patterns emerge in inter-model correlations of performance across datasets and metrics?

Key findings

• Transformer-based models (CLIP-ViT-B, CLIP-ResNet50, VisionTransformer-H/14) generally outperform CNN-based models across all datasets.
• On Geological Images, CLIP-ViT-B achieves the highest accuracy and near-top scores on macro and weighted metrics.
• On CIFAR-10, ViT-H/14 reaches the highest accuracy (~0.98), with CLIP-ViT-B and ConvNeXt-L also performing strongly.
• On STL10, ViT-H/14 and ConvNeXt-L reach top accuracies (~1.00 and ~0.99, respectively).
• CNN-only architectures show weaker and more variable performance, with AlexNet consistently underperforming across datasets.
• Correlation analyses indicate CNN-based models cluster with high inter-model similarity, while transformer-based models (CLIP variants and Vision Transformer) form a distinct performance group.

Figure 2: Examples of images included in the Geological Images dataset [ 9 ] .