[Paper Review] UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model

Jiabo Ye, Anwen Hu+12 more|arXiv (Cornell University)|Oct 8, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

UReader introduces a universal, OCR-free visually-situated language understanding framework using instruction tuning on a Multimodal Large Language Model (MLLM), achieving state-of-the-art performance across 8 out of 10 tasks without downstream finetuning. It leverages a shape-adaptive cropping module to process high-resolution images with a frozen low-resolution vision encoder and enhances text and semantic understanding via auxiliary text reading and key points generation tasks with minimal parameter updates.

ABSTRACT

Text is ubiquitous in our visual world, conveying crucial information, such as in documents, websites, and everyday photographs. In this work, we propose UReader, a first exploration of universal OCR-free visually-situated language understanding based on the Multimodal Large Language Model (MLLM). By leveraging the shallow text recognition ability of the MLLM, we only finetuned 1.2% parameters and the training cost is much lower than previous work following domain-specific pretraining and finetuning paradigms. Concretely, UReader is jointly finetuned on a wide range of Visually-situated Language Understanding tasks via a unified instruction format. To enhance the visual text and semantic understanding, we further apply two auxiliary tasks with the same format, namely text reading and key points generation tasks. We design a shape-adaptive cropping module before the encoder-decoder architecture of MLLM to leverage the frozen low-resolution vision encoder for processing high-resolution images. Without downstream finetuning, our single model achieves state-of-the-art ocr-free performance in 8 out of 10 visually-situated language understanding tasks, across 5 domains: documents, tables, charts, natural images, and webpage screenshots. Codes and instruction-tuning datasets will be released.

Motivation & Objective

• To enable universal OCR-free visually-situated language understanding across diverse image types such as documents, tables, charts, natural images, and webpages.
• To reduce training cost by avoiding domain-specific pretraining and instead using low-cost instruction tuning on a frozen MLLM.
• To improve text recognition and semantic understanding in visually-situated tasks through auxiliary instruction-tuned tasks: text reading and key points generation.
• To enable effective use of a frozen low-resolution vision encoder for high-resolution images via a shape-adaptive cropping module that preserves text integrity.
• To achieve state-of-the-art performance across 10 benchmark tasks without any downstream finetuning.

Proposed method

• Reformulate all visually-situated language understanding tasks into a unified instruction-tuning format to enable joint finetuning on a single MLLM.
• Introduce two auxiliary tasks—text reading and key points generation—using the same instruction format to enhance visual-textual and semantic understanding.
• Design a shape-adaptive cropping module that divides high-resolution images into local patches based on aspect ratio, preserving text clarity and reducing distortion.
• Apply a trainable visual abstractor to each cropped patch and concatenate their features, with learnable crop position embeddings to maintain spatial awareness.
• Feed a resized global image alongside cropped features to mitigate information loss from cropping.
• Fine-tune the MLLM with only 1.2% of parameters updated, significantly reducing training cost compared to prior domain-specific pretraining approaches.

Experimental results

Research questions

• RQ1 Can a single MLLM achieve universal OCR-free visually-situated language understanding across diverse domains without domain-specific pretraining?
• RQ2 How effective is instruction tuning with auxiliary tasks in improving text recognition and semantic comprehension in vision-language tasks?
• RQ3 Can a shape-adaptive cropping strategy effectively utilize a frozen low-resolution vision encoder for high-resolution image understanding?
• RQ4 To what extent does the model generalize across different image types, including documents, tables, charts, natural images, and webpages?
• RQ5 How does the performance of the model compare to prior methods in terms of OCR-free accuracy and training efficiency?

Key findings

• UReader achieves state-of-the-art performance in 8 out of 10 visually-situated language understanding tasks across 5 domains without any downstream finetuning.
• The shape-adaptive cropping module improves text reading performance by 3.5 BLEU points (from 21.4 to 24.9 BLEU1) compared to fixed-grid cropping, reducing text distortion.
• The auxiliary text reading task significantly enhances the model’s ability to extract and reconstruct text sequences from images, with a BLEU4 score of 11.7 on the evaluation set.
• The model demonstrates strong zero-shot generalization, accurately answering complex questions on tables, such as identifying the first movie or computing totals, by attending to relevant regions.
• UReader successfully performs open-ended key points generation on charts, though performance is limited by hallucination and incomplete understanding of complex visual structures.
• The model achieves high efficiency with only 1.2% of parameters finetuned, resulting in significantly lower training costs than prior end-to-end or two-stage approaches.