[Paper Review] LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model

Peng Gao, Jiaming Han+10 more|arXiv (Cornell University)|Apr 28, 2023

Multimodal Machine Learning Applications117 citations

TL;DR

LLaMA-Adapter V2 extends LLaMA-Adapter with bias tuning, early fusion, and disjoint-parameter joint training to enable open-ended visual instruction following using only a small amount of image-text and instruction data, while optionally integrating expert visual systems; it adds ~14M trainable parameters (~0.04% of LLaMA) to achieve strong multi-modal and language instruction performance.

ABSTRACT

How to efficiently transform large language models (LLMs) into instruction followers is recently a popular research direction, while training LLM for multi-modal reasoning remains less explored. Although the recent LLaMA-Adapter demonstrates the potential to handle visual inputs with LLMs, it still cannot generalize well to open-ended visual instructions and lags behind GPT-4. In this paper, we present LLaMA-Adapter V2, a parameter-efficient visual instruction model. Specifically, we first augment LLaMA-Adapter by unlocking more learnable parameters (e.g., norm, bias and scale), which distribute the instruction-following ability across the entire LLaMA model besides adapters. Secondly, we propose an early fusion strategy to feed visual tokens only into the early LLM layers, contributing to better visual knowledge incorporation. Thirdly, a joint training paradigm of image-text pairs and instruction-following data is introduced by optimizing disjoint groups of learnable parameters. This strategy effectively alleviates the interference between the two tasks of image-text alignment and instruction following and achieves strong multi-modal reasoning with only a small-scale image-text and instruction dataset. During inference, we incorporate additional expert models (e.g. captioning/OCR systems) into LLaMA-Adapter to further enhance its image understanding capability without incurring training costs. Compared to the original LLaMA-Adapter, our LLaMA-Adapter V2 can perform open-ended multi-modal instructions by merely introducing 14M parameters over LLaMA. The newly designed framework also exhibits stronger language-only instruction-following capabilities and even excels in chat interactions. Our code and models are available at https://github.com/ZrrSkywalker/LLaMA-Adapter.

Motivation & Objective

• Motivate building instruction-following visual models without large-scale multi-modal data.
• Introduce parameter-efficient strategies to fuse visual information into a frozen LLM.
• Propose a joint training scheme that separates image-text alignment from language instruction learning.
• Enable integration with external expert vision systems to boost visual understanding.

Proposed method

• Bias tuning of linear layers by adding trainable bias and scale to all linear modules while unfreezing normalization layers.
• Joint training with disjoint parameter groups: train visual projections and early zero-initialized attention for image-text caption data; train late adaptation prompts, gating, and additional LLaMA parameters for instruction data.
• Early fusion of visual knowledge by injecting visual tokens at early LLM layers rather than into adaptation prompts across layers.
• Incorporation of expert models (captioning/OCR/detection) during inference to enhance image understanding without extra training.
• Training on 52K image-text captions (COCO) and 567K caption data, plus 80K conversation data, with a 7B–65B LLaMA backbone.
• Moderate parameter footprint: ~14M trainable parameters, ~0.04% of the full model.

Experimental results

Research questions

• RQ1 Can LLaMA-Adapter V2 achieve open-ended visual instruction-following with limited multi-modal data and minimal parameter updates?
• RQ2 Does an early fusion strategy improve the balance between image-text alignment and language instruction tasks?
• RQ3 How does joint training with disjoint parameters affect interference between vision-language alignment and instruction following?
• RQ4 What is the impact of integrating external expert vision systems on zero-shot multimodal reasoning?

Key findings

• LLaMA-Adapter V2 surpasses its predecessor in language instruction-following and supports multi-turn dialogue.
• An early fusion strategy effectively balances visual and language fine-tuning, enabling visual instruction learning without high-quality multi-modal data.
• Disjoint-parameter joint training enables learning from image-text captions and language instructions without catastrophic interference.
• Incorporating external expert systems at inference enhances image understanding without requiring costly joint vision-language pretraining.
• With 14M trainable parameters, LLaMA-Adapter V2 achieves strong visual instruction capabilities while remaining highly parameter-efficient.