[Paper Review] MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

Deyao Zhu, Jun Chen+3 more|arXiv (Cornell University)|Apr 20, 2023

Multimodal Machine Learning ApplicationsComputer Science472 citations

TL;DR

MiniGPT-4 aligns a frozen vision encoder with a frozen advanced LLM (Vicuna) via a single projection layer, enabling GPT-4–like vision-language abilities with two-stage training and curated high-quality data.

ABSTRACT

The recent GPT-4 has demonstrated extraordinary multi-modal abilities, such as directly generating websites from handwritten text and identifying humorous elements within images. These features are rarely observed in previous vision-language models. However, the technical details behind GPT-4 continue to remain undisclosed. We believe that the enhanced multi-modal generation capabilities of GPT-4 stem from the utilization of sophisticated large language models (LLM). To examine this phenomenon, we present MiniGPT-4, which aligns a frozen visual encoder with a frozen advanced LLM, Vicuna, using one projection layer. Our work, for the first time, uncovers that properly aligning the visual features with an advanced large language model can possess numerous advanced multi-modal abilities demonstrated by GPT-4, such as detailed image description generation and website creation from hand-drawn drafts. Furthermore, we also observe other emerging capabilities in MiniGPT-4, including writing stories and poems inspired by given images, teaching users how to cook based on food photos, and so on. In our experiment, we found that the model trained on short image caption pairs could produce unnatural language outputs (e.g., repetition and fragmentation). To address this problem, we curate a detailed image description dataset in the second stage to finetune the model, which consequently improves the model's generation reliability and overall usability. Our code, pre-trained model, and collected dataset are available at https://minigpt-4.github.io/.

Motivation & Objective

• Investigate whether aligning visual features with an advanced LLM enables GPT-4–like vision-language capabilities.
• Demonstrate that training only a single projection layer can effectively fuse vision and language models.
• Show that a second-stage finetuning on high-quality image descriptions improves generation reliability and usability.

Proposed method

• Use a frozen BLIP-2 style vision encoder (ViT-G/14 with Q-Former) and a frozen Vicuna LLM as the language decoder.
• Add a single linear projection layer to align visual features with Vicuna embeddings.
• Two-stage training: (i) pretrain on large image-caption pairs with frozen components; (ii) finetune on a curated high-quality image-description dataset with a designed conversational template.
• Generate detailed image descriptions by prompting the model in a Vicuna-style conversational format and post-process with ChatGPT for quality control.
• Evaluate with qualitative demonstrations and quantitative benchmarks on advanced vision-language tasks and COCO captioning.

Figure 1: The architecture of MiniGPT-4. It consists of a vision encoder with a pretrained ViT and Q-Former, a single linear projection layer, and an advanced Vicuna large language model. MiniGPT-4 only requires training the linear projection layer to align the visual features with the Vicuna.

Experimental results

Research questions

• RQ1 Can aligning visual features with an advanced LLM enable GPT-4–level vision-language capabilities without updating the entire architecture?
• RQ2 Is a single projection layer sufficient to align vision and language models effectively under limited data?
• RQ3 Does second-stage finetuning with detailed image descriptions improve generation reliability and usability?
• RQ4 What emergent abilities does MiniGPT-4 exhibit compared to baseline vision-language models?

Key findings

• MiniGPT-4 achieves advanced abilities such as detailed image descriptions, meme interpretation, and website creation from handwritten drafts.
• A single linear projection layer suffices to align the frozen vision encoder with Vicuna, achieving GPT-4–style capabilities with ~10 hours of training on 4 A100 GPUs.
• Second-stage finetuning with a curated high-quality image-description dataset substantially reduces generation failures (e.g., detailed captions and poems) and improves language naturalness.
• On advanced tasks, MiniGPT-4 substantially outperforms BLIP-2 in user-judged responses for memes, recipes, ads, and poems (approximately 65% overall success in their qualitative test).
• For COCO captioning, MiniGPT-4 improves ground-truth coverage judgments (66.2% vs 27.5% for BLIP-2) when evaluated with ChatGPT.
• Ablation and architectural variations show that removing the Q-Former or adding layers does not improve beyond the single-projection design under limited data.

Figure 2: Detailed description