[Paper Review] Self-Supervised Learning for Fine-Grained Image Classification

Farha Al Breiki, Muhammad Ridzuan+1 more|arXiv (Cornell University)|Jul 29, 2021

Image Processing and 3D Reconstruction4 citations

TL;DR

This paper investigates self-supervised learning (SSL) for fine-grained image classification using three pretext tasks: jigsaw puzzle solving, adversarial super-resolution (SRGAN), and contrastive learning (SimCLR). It achieves a downstream validation accuracy of 83% on the cassava disease dataset, demonstrating that SSL can effectively learn discriminative features without extensive human annotations, though performance lags behind a supervised baseline of 88%.

ABSTRACT

Fine-grained image classification involves identifying different subcategories of a class which possess very subtle discriminatory features. Fine-grained datasets usually provide bounding box annotations along with class labels to aid the process of classification. However, building large scale datasets with such annotations is a mammoth task. Moreover, this extensive annotation is time-consuming and often requires expertise, which is a huge bottleneck in building large datasets. On the other hand, self-supervised learning (SSL) exploits the freely available data to generate supervisory signals which act as labels. The features learnt by performing some pretext tasks on huge unlabelled data proves to be very helpful for multiple downstream tasks. Our idea is to leverage self-supervision such that the model learns useful representations of fine-grained image classes. We experimented with 3 kinds of models: Jigsaw solving as pretext task, adversarial learning (SRGAN) and contrastive learning based (SimCLR) model. The learned features are used for downstream tasks such as fine-grained image classification. Our code is available at http://github.com/rush2406/Self-Supervised-Learning-for-Fine-grained-Image-Classification

Motivation & Objective

• To reduce reliance on expensive, expert-annotated datasets for fine-grained image classification by leveraging self-supervised learning (SSL).
• To evaluate the effectiveness of different SSL pretext tasks—jigsaw solving, adversarial super-resolution, and contrastive learning—on fine-grained classification.
• To investigate whether self-supervised representations can match or approach the performance of supervised models on fine-grained visual recognition.
• To analyze the model's ability to learn discriminative features such as texture and shape, and identify limitations such as poor color discrimination.

Proposed method

• Pretrained models were trained on 12,000 unlabelled cassava leaf images using three SSL approaches: jigsaw puzzle solving, SRGAN-based super-resolution, and SimCLR contrastive learning.
• For jigsaw, the image was split into 3×3 patches and the model predicted the correct patch arrangement, encouraging spatial and local feature learning.
• For SRGAN, the generator was trained to reconstruct high-resolution images from low-resolution inputs, with the discriminator distinguishing real from generated images.
• For SimCLR, data augmentations such as patch swapping, coarse dropout, and jigsaw were applied to create positive views for contrastive learning.
• Downstream classification was performed on 6,000 labelled images using the learned features, with class imbalance addressed via data augmentation and class weights.
• Model performance was evaluated using validation accuracy, and Grad-CAM visualizations were used to interpret learned attention and feature importance.

Experimental results

Research questions

• RQ1 Can self-supervised learning effectively learn fine-grained visual representations without human-annotated bounding boxes or part-level supervision?
• RQ2 How do different SSL pretext tasks—jigsaw, adversarial super-resolution, and contrastive learning—compare in performance on fine-grained classification?
• RQ3 To what extent can self-supervised models learn discriminative features such as texture and shape, and what limitations exist in learning color-based distinctions?
• RQ4 Does regularization via dropout improve feature learning in self-supervised models, particularly in the generator of SRGAN?
• RQ5 Can self-supervised models achieve performance close to supervised baselines in fine-grained classification tasks?

Key findings

• The SRGAN generator achieved the highest downstream classification accuracy of 83.3% when using a dropout rate of 0.5 in the classifier, outperforming other SSL methods.
• The baseline supervised model achieved 88% accuracy, indicating that while SSL is promising, it still lags behind fully supervised learning in this setting.
• The jigsaw pretext task performed the worst among the three SSL methods, suggesting limited effectiveness for capturing fine-grained distinctions in this dataset.
• Grad-CAM visualization revealed that the SRGAN generator initially failed to focus on leaf features, but after adding dropout, it learned to attend to relevant leaf regions, outperforming the discriminator.
• The models struggled to use color as a distinguishing factor, as shown in failure cases where differently shaded leaves were misclassified.
• Dropout regularization significantly improved feature learning, especially in the SRGAN generator, enabling it to surpass the discriminator in classification performance.