[Paper Review] Pixtral 12B

P. C. Agrawal, Szymon Antoniak+40 more|arXiv (Cornell University)|Oct 9, 2024

Transport Systems and Technology4 citations

TL;DR

Pixtral 12B is a 12-billion-parameter multimodal language model that excels in both text and vision tasks, using a novel vision encoder trained from scratch to process images at native resolution and aspect ratio. It outperforms larger open models like Llama-3.2 90B and closed models such as Claude-3 Haiku on multimodal benchmarks, while maintaining strong text-only performance, and is released under the Apache 2.0 license with standardized evaluation protocols.

ABSTRACT

We introduce Pixtral-12B, a 12--billion-parameter multimodal language model. Pixtral-12B is trained to understand both natural images and documents, achieving leading performance on various multimodal benchmarks, surpassing a number of larger models. Unlike many open-source models, Pixtral is also a cutting-edge text model for its size, and does not compromise on natural language performance to excel in multimodal tasks. Pixtral uses a new vision encoder trained from scratch, which allows it to ingest images at their natural resolution and aspect ratio. This gives users flexibility on the number of tokens used to process an image. Pixtral is also able to process any number of images in its long context window of 128K tokens. Pixtral 12B substanially outperforms other open models of similar sizes (Llama-3.2 11B \& Qwen-2-VL 7B). It also outperforms much larger open models like Llama-3.2 90B while being 7x smaller. We further contribute an open-source benchmark, MM-MT-Bench, for evaluating vision-language models in practical scenarios, and provide detailed analysis and code for standardized evaluation protocols for multimodal LLMs. Pixtral-12B is released under Apache 2.0 license.

Motivation & Objective

• To develop a high-performing, open-source multimodal language model that maintains strong text-only capabilities while excelling in multimodal reasoning.
• To enable flexible processing of images at variable resolutions and aspect ratios through a vision encoder trained from scratch.
• To address inconsistencies in multimodal model evaluation by standardizing prompts and metrics, reducing bias from format mismatches.
• To create a practical, multi-turn, multimodal benchmark (MM-MT-Bench) that reflects real-world assistant usage.
• To release a production-ready, open-source model with reproducible evaluation code and standardized protocols.

Proposed method

• Training a new vision encoder, Pixtral-ViT, from scratch using a novel RoPE-2D positional encoding to support variable image sizes and aspect ratios.
• Using block-diagonal attention masks to enable sequence packing and efficient batching in the vision encoder.
• Employing a 12B-parameter Mistral Nemo-based decoder for high-level multimodal reasoning and instruction following.
• Designing 'explicit' prompts that specify required answer formats to reduce evaluation bias from format mismatches.
• Introducing flexible parsing for evaluation metrics to avoid penalizing substantively correct but differently formatted answers.
• Creating and open-sourcing the MM-MT-Bench benchmark for evaluating multimodal models in long-form, multi-turn, instruction-following scenarios.

Figure 1 : Pixtral Performance. Pixtral outperforms all open-models within its weight class on multimodal tasks by a substantial margin. Left: Performance on MM-MT-Bench, a new multimodal, multiturn, instruction following benchmark designed to reflect real world usage of multimodal language models.

Experimental results

Research questions

• RQ1 Can a 12B-parameter multimodal model achieve state-of-the-art performance across both text and vision tasks without sacrificing text-only capabilities?
• RQ2 How does processing images at native resolution and aspect ratio affect multimodal reasoning performance compared to fixed-resolution methods?
• RQ3 To what extent do inconsistent evaluation prompts and exact-match metrics distort performance comparisons between multimodal models?
• RQ4 Does a multi-turn, instruction-following benchmark like MM-MT-Bench better reflect real-world utility than standard multiple-choice or short-answer benchmarks?
• RQ5 Can a standardized evaluation protocol with explicit prompts and flexible parsing improve fairness and reproducibility in multimodal model evaluation?

Key findings

• Pixtral 12B outperforms Llama-3.2 11B and Qwen2-VL 7B on multimodal benchmarks such as MMMU and MathVista, while matching or exceeding their performance on text-only tasks like MATH and HumanEval.
• Pixtral 12B surpasses the much larger Llama-3.2 90B model on multimodal benchmarks, demonstrating strong parameter efficiency.
• Pixtral 12B ranks higher than several closed-source models, including Claude-3 Opus and Claude-3 Sonnet, on the LMSys Vision Leaderboard, making it the highest-ranked Apache 2.0 model.
• The model achieves superior performance on complex chart and document understanding tasks when processing images at high resolution, outperforming CLIPA-based models that use fixed or tiled resolutions.
• Performance on MM-MT-Bench correlates highly with human preference rankings on the LMSys Vision Leaderboard, validating its practical relevance.
• The use of explicit prompts and flexible parsing significantly improves fairness and consistency in evaluation, reducing performance discrepancies caused by formatting mismatches.

Figure 2 : Pixtral Vision Encoder. Pixtral uses a new vision encoder, which is trained from scratch to natively support variable image sizes and aspect ratios. Block-diagonal attention masks enable sequence packing for batching, while RoPE-2D encodings facilitate variable image sizes. Note that the