[Paper Review] PaliGemma: A versatile 3B VLM for transfer

Lucas Beyer, Andreas Steiner+33 more|arXiv (Cornell University)|Jul 10, 2024

Advanced Fiber Optic Sensors11 citations

TL;DR

PaliGemma is a open, sub-3B vision-language model combining SigLIP-So400m image encoder with Gemma-2B decoder, trained to transfer across diverse open-world tasks including VLM benchmarks, remote sensing, and segmentation.

ABSTRACT

PaliGemma is an open Vision-Language Model (VLM) that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model. It is trained to be a versatile and broadly knowledgeable base model that is effective to transfer. It achieves strong performance on a wide variety of open-world tasks. We evaluate PaliGemma on almost 40 diverse tasks including standard VLM benchmarks, but also more specialized tasks such as remote-sensing and segmentation.

Motivation & Objective

• Provide a versatile base vision-language model that transfers well across a broad set of tasks.
• Show that a small, open VLM can match or exceed larger models on standard and niche tasks.
• Demonstrate effective multi-stage pretraining and transfer strategies for diverse outputs.
• Investigate design choices that influence multimodal pretraining and transfer performance.
• Offer practical transfer recipes with guidelines for hyper-parameters and resolution handling.

Proposed method

• Assemble a three-component VLM: SigLIP image encoder, Gemma-2B decoder LM, and a linear projection connecting image tokens to language-model inputs.
• Train via a staged process: Stage0 unimodal pretraining, Stage1 multimodal pretraining on a broad task mix with no freezing, Stage2 higher-resolution finetuning, Stage3 task-specific transfer to diverse benchmarks.
• Use prefix-LM masking to allow full attention on input (image and prefix) with autoregressive suffix generation, and train with supervision only on the suffix.
• Extend Gemma vocabulary with 1024 location tokens and 128 segmentation tokens, initialize new tokens carefully to optimize transfer.
• Employ a simple linear connector (vs. MLP) and avoid freezing the image encoder during Stage1 for better spatial understanding and transferability.
• Adopt a unified transfer recipe with task-specific hyperparameters (resolution, epochs, learning rate, label smoothing, dropout, etc.) across 30+ academic benchmarks, ensuring images are de-duplicated from pretraining data.

Figure 1: PaliGemma’s architecture: a SigLIP image encoder feeds into a Gemma decoder LM.

Experimental results

Research questions

• RQ1 How does a compact open VLM (≈3B params) perform across a wide spectrum of vision-language tasks relative to larger models?
• RQ2 What pretraining and training-design choices most strongly influence transfer performance for open VLMs?
• RQ3 Can higher image resolution during transfer be achieved effectively via staged pretraining without exponential data or compute or instruction tuning?
• RQ4 What are practical transfer recipes (hyperparameters, augmentation, architecture choices) that yield strong results across standard VLM benchmarks and specialized tasks (remote sensing, infographic QA, segmentation, video tasks)?

Key findings

• PaliGemma (3B scale) achieves competitive performance on standard VLM benchmarks (e.g., COCO, VQA, TextVQA) and on specialized tasks (remote sensing VQA, infographic QA, video captioning, referring expression segmentation).
• A 3-stage pretraining schedule (unimodal, multimodal with an unfrozen image encoder, then resolution increase) yields broad transferability to diverse tasks.
• Prefix-LM masking with input full attention and suffix autoregression outperforms alternatives; task prefixes help for ambiguous tasks and suffix-only supervision is effective.
• Adding new tokens (location and segmentation tokens) requires careful initialization (AvgEmb can hurt long-term transfer), with standard Gaussian initialization yielding better transfer.
• Not freezing the image encoder during Stage1 improves spatial understanding and transferability, while freezing the LLM harms performance; linear connectors outperform MLPs in this setup.
• Providing multiple resolution checkpoints (224, 448, 896) and upcycling through Stage2 is beneficial, with disentangled effects from higher resolution content vs. longer sequence length.

Figure 2: PaliGemma’s Prefix-LM masking: block attention throughout image and prefix, autoregressive attention on the suffix. Each square indicates whether the row can attend to the column.