[Paper Review] How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites

Zhe Chen, Weiyun Wang+33 more|arXiv (Cornell University)|Apr 25, 2024

Reservoir Engineering and Simulation Methods16 citations

TL;DR

InternVL 1.5 is an open-source multimodal large language model that closes the gap to commercial models like GPT-4V by enhancing vision encoding, enabling high-resolution input, and expanding bilingual data, achieving state-of-the-art results on several benchmarks.

ABSTRACT

In this report, we introduce InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple improvements: (1) Strong Vision Encoder: we explored a continuous learning strategy for the large-scale vision foundation model -- InternViT-6B, boosting its visual understanding capabilities, and making it can be transferred and reused in different LLMs. (2) Dynamic High-Resolution: we divide images into tiles ranging from 1 to 40 of 448$ imes$448 pixels according to the aspect ratio and resolution of the input images, which supports up to 4K resolution input. (3) High-Quality Bilingual Dataset: we carefully collected a high-quality bilingual dataset that covers common scenes, document images, and annotated them with English and Chinese question-answer pairs, significantly enhancing performance in OCR- and Chinese-related tasks. We evaluate InternVL 1.5 through a series of benchmarks and comparative studies. Compared to both open-source and proprietary models, InternVL 1.5 shows competitive performance, achieving state-of-the-art results in 8 of 18 benchmarks. Code has been released at https://github.com/OpenGVLab/InternVL.

Motivation & Objective

• Bridge the capability gap between open-source and proprietary multimodal models in multimodal understanding.
• Enhance visual feature extraction with a stronger vision encoder through continuous learning of InternViT-6B.
• Enable high-resolution, tile-based image processing up to 4K while maintaining efficiency.
• Improve bilingual (English-Chinese) multimodal performance via high-quality bilingual datasets and translation pipelines.

Proposed method

• Adopt ViT-MLP-LLM architecture by integrating a strong vision encoder (InternViT-6B) with an LLM (InternLM2-20B) via an MLP projector.
• Implement dynamic high-resolution input by tiling images into 448x448 patches (1–12 tiles during training; up to 40 tiles during testing) to reach 4K resolution.
• Pre-train with a diverse, high-quality bilingual dataset covering English and Chinese annotations for OCR and other multimodal tasks.
• Use a data translation pipeline to convert English datasets into Chinese (and potentially other languages) to boost multilingual capabilities.
• Fine-tune the entire model (26B parameters) after initial visual and projection training to optimize multimodal performance.

Experimental results

Research questions

• RQ1 How close can an open-source MLLM approach the performance of leading commercial models across OCR, general multimodal, math, and multi-turn conversation benchmarks?
• RQ2 What combination of vision encoder strength, high-resolution dynamic input, and bilingual data quality yields the largest gains in multimodal understanding and bilingual capabilities?
• RQ3 Can an open-source model with these improvements outperform competitors on document and OCR-centric tasks while maintaining robust multilingual performance?

Key findings

• InternVL 1.5 achieves competitive performance with open-source and proprietary models across 18 multimodal benchmarks.
• The model attains state-of-the-art results in 8 of 18 benchmarks, including OCR-related tasks like ChartQA and OCRBench.
• In OCR and document-oriented tasks, InternVL 1.5 can surpass leading commercial models on several datasets, and exhibits strong Chinese language capabilities.
• Dynamic high-resolution processing enables near 4K input handling without prohibitive compute, while maintaining robust performance at lower tile counts.
• Continuous learning of the vision encoder (InternViT-6B) enhances visual representation to levels comparable to much larger LLMs in practice.
• In math reasoning tasks, InternVL 1.5 outperforms several competitors, including GPT-4V, on MathVista-related benchmarks.