[Paper Review] LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images

Ruyi Xu, Yuan Yao+8 more|arXiv (Cornell University)|Mar 18, 2024

Medical Imaging Techniques and ApplicationsMedicine3 citations

TL;DR

LLaVA-UHD is a large multimodal model that enables efficient perception of images in any aspect ratio and high resolution through a modularized image encoding strategy, compression of visual tokens, and spatial token organization. It achieves 6.4 higher TextVQA accuracy and supports 672×1088 resolution with 94% of LLaVA-1.5's computation, outperforming models trained on 2–3× more data.

ABSTRACT

Visual encoding constitutes the basis of large multimodal models (LMMs) in understanding the visual world. Conventional LMMs process images in fixed sizes and limited resolutions, while recent explorations in this direction are limited in adaptivity, efficiency, and even correctness. In this work, we first take GPT-4V and LLaVA-1.5 as representative examples and expose systematic flaws rooted in their visual encoding strategy. To address the challenges, we present LLaVA-UHD, a large multimodal model that can efficiently perceive images in any aspect ratio and high resolution. LLaVA-UHD includes three key components: (1) An image modularization strategy that divides native-resolution images into smaller variable-sized slices for efficient and extensible encoding, (2) a compression module that further condenses image tokens from visual encoders, and (3) a spatial schema to organize slice tokens for LLMs. Comprehensive experiments show that LLaVA-UHD outperforms established LMMs trained with 2-3 orders of magnitude more data on 9 benchmarks. Notably, our model built on LLaVA-1.5 336x336 supports 6 times larger (i.e., 672x1088) resolution images using only 94% inference computation, and achieves 6.4 accuracy improvement on TextVQA. Moreover, the model can be efficiently trained in academic settings, within 23 hours on 8 A100 GPUs (vs. 26 hours of LLaVA-1.5). We make the data and code publicly available at https://github.com/thunlp/LLaVA-UHD.

Motivation & Objective

• To address the limitations of existing LMMs in handling high-resolution and arbitrary aspect ratio images due to fixed-size and fixed-ratio visual encoding.
• To investigate and expose systematic flaws in visual encoding strategies of leading models like GPT-4V and LLaVA-1.5 that lead to correctness issues and hallucinations.
• To develop a scalable, efficient, and adaptive LMM architecture that preserves visual encoder capabilities while supporting diverse image inputs.
• To enable effective instruction tuning and training within academic constraints, reducing training time compared to baseline models.

Proposed method

• Introduces an image modularization strategy that divides native-resolution images into variable-sized, non-overlapping slices to avoid padding and distortion.
• Employs a compression module to reduce the number of visual tokens from the vision encoder, minimizing LLM inference cost.
• Applies a spatial schema to encode the relative positions of image slices, enabling the LLM to reason about spatial layout.
• Uses a perceiver resampler to aggregate visual features from compressed slice tokens and feed them into the LLM.
• Trains the model end-to-end with instruction tuning on vision encoder parameters, enabling adaptation to diverse image distributions.
• Supports high-resolution input by avoiding fixed-size constraints and maintaining compatibility with pre-trained CLIP-ViT encoders.

Figure 1: Experimental results of GPT-4V in identifying numbers of objects. Note that the dashed lines in (a) are for illustration purposes only, and not presented to GPT-4V.

Experimental results

Research questions

• RQ1 What are the root causes of correctness failures in high-resolution image understanding for models like GPT-4V and LLaVA-1.5?
• RQ2 Can a modularized image encoding strategy enable efficient and distortion-free processing of arbitrary aspect ratio and high-resolution images?
• RQ3 How does compressing and spatially organizing image slice tokens affect downstream multimodal reasoning performance?
• RQ4 To what extent can instruction tuning on vision encoder parameters improve generalization across diverse image inputs?
• RQ5 Can the model achieve state-of-the-art performance with significantly less training data and computation than existing LMMs?

Key findings

• LLaVA-UHD achieves a 6.4-point accuracy gain on TextVQA compared to LLaVA-1.5, demonstrating superior fine-grained visual understanding.
• The model supports 672×1088 resolution images using only 94% of the inference computation required by LLaVA-1.5, enabling higher resolution with near-constant cost.
• It outperforms established LMMs trained on 2–3 orders of magnitude more data across 9 benchmarks, indicating high data efficiency.
• The model reduces training time to 23 hours on 8 A100 GPUs, compared to 26 hours for LLaVA-1.5, showing improved training efficiency.
• The model maintains strong performance on extreme aspect ratios, with performance gains increasing as image aspect ratio deviates further from square.
• Mechanistic analysis reveals that visual encoding strategy is a key source of hallucinations and correctness failures in models like GPT-4V and LLaVA-1.5.

Figure 2: Results on probing GPT-4V via continuously changing image resolutions.