[Paper Review] Self-Supervised Backbone Framework for Diverse Agricultural Vision Tasks

Sudhir Sornapudi, Rajhans Singh|arXiv (Cornell University)|Mar 22, 2024

Smart Agriculture and AIAgricultural and Biological Sciences3 citations

TL;DR

This paper proposes a self-supervised contrastive learning framework using SimCLR to pre-train a ResNet-50 backbone on a large, unannotated agricultural field image dataset, enabling strong performance across diverse downstream vision tasks with minimal labeled data. The approach achieves 80.2% accuracy using only 1% of labeled data, significantly reducing annotation costs while improving data efficiency, transfer learning, and model convergence.

ABSTRACT

Computer vision in agriculture is game-changing with its ability to transform farming into a data-driven, precise, and sustainable industry. Deep learning has empowered agriculture vision to analyze vast, complex visual data, but heavily rely on the availability of large annotated datasets. This remains a bottleneck as manual labeling is error-prone, time-consuming, and expensive. The lack of efficient labeling approaches inspired us to consider self-supervised learning as a paradigm shift, learning meaningful feature representations from raw agricultural image data. In this work, we explore how self-supervised representation learning unlocks the potential applicability to diverse agriculture vision tasks by eliminating the need for large-scale annotated datasets. We propose a lightweight framework utilizing SimCLR, a contrastive learning approach, to pre-train a ResNet-50 backbone on a large, unannotated dataset of real-world agriculture field images. Our experimental analysis and results indicate that the model learns robust features applicable to a broad range of downstream agriculture tasks discussed in the paper. Additionally, the reduced reliance on annotated data makes our approach more cost-effective and accessible, paving the way for broader adoption of computer vision in agriculture.

Motivation & Objective

• To reduce reliance on expensive, time-consuming manual labeling in agricultural computer vision by leveraging self-supervised representation learning.
• To evaluate the effectiveness of self-supervised pretraining on diverse agricultural vision tasks such as classification, detection, segmentation, and video analysis.
• To demonstrate that domain-specific pretraining on agricultural imagery outperforms general-domain pretraining for downstream transfer learning.
• To explore practical applications of self-supervised features, including outlier detection, image retrieval, and image reconstruction.
• To develop and release PixelAffinity, a web-based content-based image retrieval tool using self-supervised features for large agroecology databases.

Proposed method

• Pre-train a ResNet-50 backbone using SimCLR contrastive learning on a large, unannotated dataset of real-world agricultural field images from mobile phones and satellites.
• Apply data augmentation strategies such as random cropping, color jittering, and Gaussian blur to generate positive sample pairs for contrastive learning.
• Fine-tune the pre-trained backbone on downstream tasks using only a small fraction of labeled data, including 1% for evaluation.
• Utilize the learned features for downstream tasks such as image classification, object detection, semantic segmentation, and video frame selection.
• Develop PixelAffinity, a web-based content-based image retrieval system that uses self-supervised features to search for semantically similar images in large agricultural databases.
• Explore image reconstruction and editing using the pre-trained encoder as a guidance network, employing GANs, PixelCNN, and diffusion models with perceptual loss functions.

Figure 1 : Wide range of applications based on representations learned from self-supervised modeling.

Experimental results

Research questions

• RQ1 Can self-supervised representation learning on unannotated agricultural field images produce robust features that generalize across diverse downstream vision tasks?
• RQ2 How does pretraining on domain-specific agricultural data compare to general-domain pretraining in terms of transfer learning performance?
• RQ3 To what extent can self-supervised models reduce the need for labeled data in agricultural computer vision, especially with minimal supervision?
• RQ4 Can self-supervised features effectively support advanced applications such as outlier detection, video frame selection, and content-based image retrieval?
• RQ5 How effective are self-supervised features in guiding image reconstruction and editing tasks in agricultural imagery?

Key findings

• Fine-tuning the self-supervised ResNet-50 backbone with only 1% of labeled data achieved 80.2% accuracy on downstream tasks, demonstrating high data efficiency.
• Models pre-trained on agricultural data outperformed those pre-trained on general-domain datasets, highlighting the importance of domain-specific pretraining.
• The self-supervised approach enabled faster model convergence during fine-tuning compared to training from random initialization.
• Clustering analysis using self-supervised features successfully identified anomalies and outliers in agricultural datasets.
• The PixelAffinity tool, built using self-supervised features, enabled efficient content-based image retrieval across large-scale agroecology databases.
• Self-supervised features proved effective for video preprocessing, enabling identification of redundant frames and selection of key frames for downstream use.

Figure 2 : Illustration of the two-stage process: self-supervised pre-training by contrasting different views of the same image and label-light supervised fine-tuning for downstream tasks.