[Paper Review] Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning

Shuo Yang, Zirui Shang+5 more|ArXiv.org|Mar 2, 2024

Text and Document Classification Technologies4 citations

TL;DR

This paper proposes a data-free framework for multi-label image recognition that leverages pre-trained LLMs (e.g., ChatGLM) to generate rich textual descriptions of object attributes and relationships, which are used to learn hierarchical prompts for fine-tuning CLIP without any training data. The method achieves state-of-the-art performance, outperforming zero-shot methods by 4.7% mAP on MS-COCO, demonstrating the effectiveness of LLM-powered prompt tuning in data-scarce scenarios.

ABSTRACT

This paper proposes a novel framework for multi-label image recognition without any training data, called data-free framework, which uses knowledge of pre-trained Large Language Model (LLM) to learn prompts to adapt pretrained Vision-Language Model (VLM) like CLIP to multilabel classification. Through asking LLM by well-designed questions, we acquire comprehensive knowledge about characteristics and contexts of objects, which provides valuable text descriptions for learning prompts. Then we propose a hierarchical prompt learning method by taking the multi-label dependency into consideration, wherein a subset of category-specific prompt tokens are shared when the corresponding objects exhibit similar attributes or are more likely to co-occur. Benefiting from the remarkable alignment between visual and linguistic semantics of CLIP, the hierarchical prompts learned from text descriptions are applied to perform classification of images during inference. Our framework presents a new way to explore the synergies between multiple pre-trained models for novel category recognition. Extensive experiments on three public datasets (MS-COCO, VOC2007, and NUS-WIDE) demonstrate that our method achieves better results than the state-of-the-art methods, especially outperforming the zero-shot multi-label recognition methods by 4.7% in mAP on MS-COCO.

Motivation & Objective

• To address the challenge of recognizing novel object categories in multi-label image recognition when no training data is available.
• To explore synergies between pre-trained vision-language models (CLIP) and large language models (LLMs) for zero-shot adaptation.
• To improve multi-label recognition performance by incorporating object relationships and shared attributes into prompt learning.
• To develop a prompt tuning strategy that leverages LLM-generated knowledge without requiring annotated images.

Proposed method

• Design and pose diverse question types to LLMs (e.g., category-agnostic, category-specific, relationship-based) to extract comprehensive attribute and contextual knowledge of objects.
• Generate rich text descriptions of object attributes (e.g., shape, color, material) and inter-object relationships (e.g., co-occurrence in scenes) using LLMs as surrogate training data.
• Propose a hierarchical prompt learning method with three token types: global shared tokens, partially shared tokens for co-occurring/similar categories, and category-specific tokens.
• Train the hierarchical prompts using the LLM-generated text descriptions as supervision, leveraging CLIP’s visual-linguistic alignment for inference.
• Integrate global and local prompt branches with learnable weights, optimizing the balance between shared and specific knowledge.
• Apply the learned prompts to classify images at inference time without any fine-tuning on real image data.

Experimental results

Research questions

• RQ1 Can LLM-generated textual knowledge effectively replace real training data for prompt tuning in multi-label image recognition?
• RQ2 How can object relationships and shared attributes be effectively encoded into learnable prompts to improve multi-label classification?
• RQ3 What is the optimal hierarchical structure of prompt tokens (global, partially shared, category-specific) for capturing multi-label dependencies?
• RQ4 How does the performance of LLM-powered prompt tuning compare to methods using human-written captions or real training data?

Key findings

• The proposed method achieves 66.8 mAP on MS-COCO, outperforming the previous SOTA zero-shot method by 4.7% mAP, demonstrating strong data-free performance.
• The method surpasses all unsupervised and unlabeled training baselines on VOC2007, MS-COCO, and NUS-WIDE, confirming the value of LLM-generated knowledge.
• The performance is robust to variations in the number of prompt tokens, with optimal performance achieved at 32 total tokens, beyond which performance degrades.
• The optimal balance between global and local prompt branches is achieved at a weight of 0.65 for global prompts, indicating their dominant yet complementary role.
• Qualitative analysis shows that hierarchical prompts improve predictions on small or ambiguous objects, though hand-crafted prompts still outperform in specific cases due to human prior knowledge.
• The method performs slightly worse than fully supervised methods, likely due to domain gaps between CLIP’s pretraining data and downstream tasks.