[Paper Review] MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning

Fuxiao Liu, Xiaoyang Wang+6 more|arXiv (Cornell University)|Nov 15, 2023

Speech and dialogue systems4 citations

TL;DR

This paper introduces MMC-Instruction, a 600k-instance instruction-tuning dataset for multimodal chart understanding, and MMCA, a vision-language model finetuned on this data that achieves state-of-the-art performance on chart QA benchmarks. It also proposes MMC-Benchmark, a human-annotated, comprehensive evaluation suite with nine diverse tasks, revealing significant limitations in existing LMMs—including GPT-4V—especially on complex tasks like chart-to-datatable and chart-to-json conversion.

ABSTRACT

With the rapid development of large language models (LLMs) and their integration into large multimodal models (LMMs), there has been impressive progress in zero-shot completion of user-oriented vision-language tasks. However, a gap remains in the domain of chart image understanding due to the distinct abstract components in charts. To address this, we introduce a large-scale MultiModal Chart Instruction ( extbf{MMC-Instruction}) dataset comprising 600k instances supporting diverse tasks and chart types. Leveraging this data, we develop MultiModal Chart Assistant ( extbf{MMCA}), an LMM that achieves state-of-the-art performance on existing chart QA benchmarks. Recognizing the need for a comprehensive evaluation of LMM chart understanding, we also propose a MultiModal Chart Benchmark ( extbf{MMC-Benchmark}), a comprehensive human-annotated benchmark with nine distinct tasks evaluating reasoning capabilities over charts. Extensive experiments on MMC-Benchmark reveal the limitations of existing LMMs on correctly interpreting charts, even for the most recent GPT-4V model. Our work provides an instruction-tuning methodology and benchmark to advance multimodal understanding of charts. Code and data are available at https://github.com/FuxiaoLiu/MMC.

Motivation & Objective

• Address the gap in large multimodal models' (LMMs) ability to understand complex, abstract chart elements such as trend lines, legends, and axis labels.
• Overcome the limitations of existing chart understanding datasets, which are often small, templated, or lack diversity in language style and chart types.
• Develop a comprehensive, human-annotated benchmark (MMC-Benchmark) to evaluate LMMs across diverse, cognition-aligned tasks beyond simple QA.
• Improve LMM performance on chart understanding through large-scale instruction tuning using a diverse, open-ended instruction dataset.
• Identify and analyze persistent weaknesses in current LMMs—especially on data extraction tasks like chart-to-datatable and chart-to-json—despite their strong general vision-language capabilities.

Proposed method

• Construct MMC-Instruction, a 600k-instance dataset using GPT-4 to generate diverse, free-form, open-ended instructions across 9 distinct chart understanding tasks.
• Design MMC-Benchmark as a human-annotated, comprehensive evaluation suite with 9 subtasks, including chart information extraction, reasoning, topic classification, and data format conversion (e.g., chart-to-datatable, chart-to-json).
• Develop MMCA, a modular, instruction-tuned LMM finetuned on MMC-Instruction using unified instruction tuning with updated vision projectors to improve visual-textual alignment.
• Employ two evaluation protocols: free-form generation evaluation using ChatGPT and multiple-choice QA to assess both reasoning and factual accuracy.
• Use a unified training strategy that updates the vision projector during fine-tuning to better integrate visual features into the LLM’s input space.
• Conduct extensive ablation and comparison experiments on both standard benchmarks (e.g., ChartQA) and MMC-Benchmark to evaluate performance across diverse chart types and tasks.

Figure 1: Diagram of our human-annotated MMC-Benchmark , consisting of 9 distinct tasks, various topics ( business, health, biology, engineering, etc ), free-form questions, open-ended answers, and unbounded plot types.

Experimental results

Research questions

• RQ1 Can large-scale, diverse instruction tuning significantly improve LMMs’ performance on multimodal chart understanding beyond zero-shot or few-shot capabilities?
• RQ2 How do existing LMMs, including GPT-4V, perform on complex, open-ended chart understanding tasks such as chart-to-datatable and chart-to-json conversion?
• RQ3 To what extent does the proposed MMC-Benchmark reveal limitations in current LMMs that are not exposed by existing, simpler benchmarks?
• RQ4 What are the key failure modes of current LMMs in chart understanding, particularly regarding instruction following and visual feature integration?
• RQ5 Can a unified instruction-tuning approach with a large, diverse dataset like MMC-Instruction close the performance gap between open-source LMMs and proprietary models like GPT-4V on chart reasoning tasks?

Key findings

• MMCA achieves state-of-the-art performance on existing chart QA benchmarks, significantly outperforming existing open-source LMMs across all nine tasks in MMC-Benchmark.
• Even GPT-4V, the most advanced multimodal model tested, performs poorly on chart-to-datatable and chart-to-json tasks, indicating a critical gap in data extraction accuracy.
• The overall scores of existing LMMs on MMC-Benchmark are substantially lower than on previous benchmarks like ChartQA, due to the increased diversity and open-ended nature of questions and answers.
• The MMC-Instruction dataset enables a large performance boost over baseline LMMs across all nine tasks in MMC-Benchmark, demonstrating its effectiveness for instruction tuning.
• Common failure modes include poor instruction following (e.g., answering 'Yes' to a proportion question), weak vision encoders (e.g., CLIP-based models), and inaccurate OCR-like data extraction.
• Vision encoder limitations and lack of fine-tuning on chart-specific data are major contributors to poor performance, especially on tasks requiring precise data value extraction.

Figure 2: The overall architecture of MMCA , which is continuously trained in two stages.