[Paper Review] Leveraging Vision Language Models for Specialized Agricultural Tasks

Muhammad Arshad, Talukder Z. Jubery+9 more|arXiv (Cornell University)|Jul 29, 2024

Smart Agriculture and AIAgricultural and Biological Sciences3 citations

TL;DR

This paper introduces AgEval, a benchmark for evaluating multimodal large language models (LLMs) on zero-shot and few-shot plant stress phenotyping tasks across 12 diverse agricultural challenges. It demonstrates that few-shot in-context learning significantly boosts performance—especially with relevant (bullseye) examples—while highlighting substantial variability in model consistency across classes, with GPT-4o showing the strongest overall adaptability and F1 scores rising from 46.24% to 73.37% in 8-shot settings.

ABSTRACT

As Vision Language Models (VLMs) become increasingly accessible to farmers and agricultural experts, there is a growing need to evaluate their potential in specialized tasks. We present AgEval, a comprehensive benchmark for assessing VLMs' capabilities in plant stress phenotyping, offering a solution to the challenge of limited annotated data in agriculture. Our study explores how general-purpose VLMs can be leveraged for domain-specific tasks with only a few annotated examples, providing insights into their behavior and adaptability. AgEval encompasses 12 diverse plant stress phenotyping tasks, evaluating zero-shot and few-shot in-context learning performance of state-of-the-art models including Claude, GPT, Gemini, and LLaVA. Our results demonstrate VLMs' rapid adaptability to specialized tasks, with the best-performing model showing an increase in F1 scores from 46.24% to 73.37% in 8-shot identification. To quantify performance disparities across classes, we introduce metrics such as the coefficient of variation (CV), revealing that VLMs' training impacts classes differently, with CV ranging from 26.02% to 58.03%. We also find that strategic example selection enhances model reliability, with exact category examples improving F1 scores by 15.38% on average. AgEval establishes a framework for assessing VLMs in agricultural applications, offering valuable benchmarks for future evaluations. Our findings suggest that VLMs, with minimal few-shot examples, show promise as a viable alternative to traditional specialized models in plant stress phenotyping, while also highlighting areas for further refinement. Results and benchmark details are available at: https://github.com/arbab-ml/AgEval

Motivation & Objective

• To address the scalability limitations of traditional plant stress phenotyping, which relies on expert labor and is time-consuming and subjective.
• To evaluate the potential of multimodal LLMs in zero-shot and few-shot settings for agricultural tasks with limited annotated data.
• To establish a standardized benchmark—AgEval—for assessing model performance across diverse plant stress phenotyping tasks, including identification, classification, and quantification.
• To quantify the impact of example relevance and model consistency across different plant stress classes and datasets.
• To provide baseline metrics and insights for future development of domain-adapted LLMs in agriculture.

Proposed method

• The authors curated a benchmark dataset from public and internal sources, comprising 12 diverse plant stress phenotyping tasks across multiple crops and stress types.
• Tasks include identification (presence of stress), classification (type of stress), and quantification (severity or extent of stress), reflecting real-world agricultural challenges.
• The evaluation framework assesses zero-shot and few-shot in-context learning performance using state-of-the-art multimodal LLMs, including GPT-4o, Claude-3, Gemini, and LLaVA.
• Performance is measured using F1 scores and Mean Reciprocal Rank (MRR), with a focus on example relevance and inter-class consistency.
• The Coefficient of Variation (CV) is used to quantify performance uniformity across classes within each dataset, revealing model reliability across diverse stress types.
• The study analyzes the impact of different few-shot examples—bullseye (same class), related (different class), and unrelated—on prediction accuracy.

Experimental results

Research questions

• RQ1 How do state-of-the-art multimodal LLMs perform on zero-shot and few-shot plant stress phenotyping tasks?
• RQ2 What is the impact of example relevance—specifically, bullseye vs. related vs. unrelated examples—on few-shot learning performance?
• RQ3 How consistent are model predictions across different classes within each phenotyping task, and what factors contribute to performance variability?
• RQ4 Which model architecture demonstrates the strongest overall adaptability and consistency across diverse agricultural stress phenotyping tasks?
• RQ5 How do different models compare in their strengths across identification, classification, and quantification subtasks?

Key findings

• GPT-4o achieved the highest performance improvement with few-shot learning, increasing F1 scores from 46.24% (zero-shot) to 73.37% (8-shot) in identification tasks.
• The inclusion of exact category (bullseye) examples improved F1 scores by an average of 15.38%, while examples from other classes had negligible or negative impacts.
• Model consistency varied significantly across tasks, with the Coefficient of Variation (CV) ranging from 26.02% (GPT-4o) to 58.03% (Claude-3-haiku), indicating high variability in performance across classes.
• Soybean Diseases showed the highest CV (81.29%), likely due to low image resolution, while Durum Wheat had the lowest (14.28%), indicating more uniform performance.
• Gemini-pro-1.5 led in zero-shot identification (MRR = 0.69), while GPT-4o outperformed others in zero-shot classification and quantification (MRR = 0.70).
• Claude-3.5-sonnet demonstrated consistent superiority in classification and quantification tasks, highlighting model-specific strengths across task types.