[Paper Review] GIST: Generating Image-Specific Text for Fine-grained Object Classification

Kathleen M. Lewis, Emily Mu+2 more|arXiv (Cornell University)|Jul 21, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

GIST proposes a method to generate fine-grained, image-specific text descriptions for fine-grained object classification using domain-specific prompts to a large language model (GPT) and matching these to images via CLIP. The resulting image-text pairs are used to fine-tune CLIP with contrastive learning, achieving state-of-the-art performance across four diverse datasets in both full-shot and few-shot settings, with a 4.1% average accuracy gain over CLIP linear probes.

ABSTRACT

Recent vision-language models outperform vision-only models on many image classification tasks. However, because of the absence of paired text/image descriptions, it remains difficult to fine-tune these models for fine-grained image classification. In this work, we propose a method, GIST, for generating image-specific fine-grained text descriptions from image-only datasets, and show that these text descriptions can be used to improve classification. Key parts of our method include 1. prompting a pretrained large language model with domain-specific prompts to generate diverse fine-grained text descriptions for each class and 2. using a pretrained vision-language model to match each image to label-preserving text descriptions that capture relevant visual features in the image. We demonstrate the utility of GIST by fine-tuning vision-language models on the image-and-generated-text pairs to learn an aligned vision-language representation space for improved classification. We evaluate our learned representation space in full-shot and few-shot scenarios across four diverse fine-grained classification datasets, each from a different domain. Our method achieves an average improvement of $4.1\%$ in accuracy over CLIP linear probes and an average of $1.1\%$ improvement in accuracy over the previous state-of-the-art image-text classification method on the full-shot datasets. Our method achieves similar improvements across few-shot regimes. Code is available at https://github.com/emu1729/GIST.

Motivation & Objective

• To address the challenge of limited paired image-text data for fine-tuning vision-language models on fine-grained classification tasks.
• To generate diverse, class-specific, and fine-grained textual descriptions using large language models with domain-specific prompting.
• To match each image to the most relevant, label-preserving text description using a pretrained vision-language model (CLIP).
• To improve zero-shot and few-shot classification accuracy by fine-tuning CLIP on automatically generated image-text pairs.
• To demonstrate consistent performance gains across diverse domains, including dermatology, birds, flowers, and aircraft.

Proposed method

• Use domain-specific prompts (e.g., 'What does a male Western Meadowlark look like?') to generate diverse, fine-grained captions for each class using a large language model (GPT).
• Apply a pretrained vision-language model (CLIP) to match each training image to the most relevant, label-preserving caption based on visual-semantic similarity.
• Summarize the matched captions into concise, informative descriptions suitable for fine-tuning CLIP.
• Fine-tune CLIP using contrastive learning on the generated image-text pairs to align the vision and language representation spaces.
• Use the resulting aligned representation for both full-shot and few-shot image classification.
• Compare the performance of fine-tuned CLIP with alternative methods, including visual grounding models like GIT.

Experimental results

Research questions

• RQ1 Can domain-specific prompting of a large language model generate more discriminative and fine-grained captions than generic prompts?
• RQ2 Does matching image-specific captions to images via CLIP improve downstream fine-grained classification performance?
• RQ3 How does the number of captions per image and caption length affect classification accuracy?
• RQ4 Can GIST-generated image-text pairs consistently improve performance across diverse fine-grained datasets in both full-shot and few-shot scenarios?
• RQ5 How does fine-tuning CLIP on GIST-generated pairs compare to fine-tuning a visual grounding model like GIT on the same data?

Key findings

• GIST achieves an average accuracy improvement of 4.1% over CLIP linear probes on full-shot datasets across four fine-grained classification benchmarks.
• GIST improves accuracy by 1.1% on average over the previous state-of-the-art image-text classification method on full-shot datasets.
• The method achieves similar performance gains in few-shot regimes, with a 44.10% top-1 accuracy on 5-shot Fitzpatrick40 compared to 24.02% for GIT.
• Shortening the original GPT-generated captions improves downstream classification performance compared to using longer, unedited descriptions.
• Fine-tuning CLIP on GIST-generated pairs outperforms fine-tuning the visual grounding model GIT on the same image-text pairs, especially in low-data regimes.
• The method is robust across diverse domains, including dermatology (Fitzpatrick40), birds (CUB200-2011), flowers (Flowers102), and aircraft (FGVC-Aircraft).